[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:19 dist-test-slave-5rg1 krb5kdc[31511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:19 dist-test-slave-5rg1 krb5kdc[31511](info): set up 2 sockets
Sep 28 16:44:19 dist-test-slave-5rg1 krb5kdc[31511](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 16:44:20.748133 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:20 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:20.892694 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38301
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.020634 31524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.020982 31524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.021071 31524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.025315 31524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:21.025429 31524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.025463 31524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.025506 31524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:21.025545 31524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:21.032063 31524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:21.033458 31524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.034583 31524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.044426 31529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.044433 31530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.044492 31524 server_base.cc:1034] running on GCE node
W20260928 16:44:21.044433 31532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.045362 31524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.046798 31524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.048168 31524 hybrid_clock.cc:648] HybridClock initialized: now 1790613861048053 us; error 130 us; skew 500 ppm
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.051570 31524 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:21.052839 31524 webserver.cc:460] Webserver started at http://127.30.196.62:35415/ using document root <none> and password file <none>
I20260928 16:44:21.053499 31524 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.053609 31524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.053954 31524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.056425 31524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "27e6eb0333fc4cff85b57c108a85350d"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "b09a03dcd45b04aecaf0b55266ad1cc2"
server_key_iv: "e29ba42005251e4c097e8ef4d9272d03"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.057057 31524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "27e6eb0333fc4cff85b57c108a85350d"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "b09a03dcd45b04aecaf0b55266ad1cc2"
server_key_iv: "e29ba42005251e4c097e8ef4d9272d03"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.062239 31524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:21.065332 31539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.066859 31524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:21.067046 31524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "27e6eb0333fc4cff85b57c108a85350d"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "b09a03dcd45b04aecaf0b55266ad1cc2"
server_key_iv: "e29ba42005251e4c097e8ef4d9272d03"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.067194 31524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.090735 31524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.095000 31524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.095255 31524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.103678 31524 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38301
I20260928 16:44:21.103746 31592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38301 every 8 connection(s)
I20260928 16:44:21.104807 31524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:21.108152 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:21.113101 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 31524
I20260928 16:44:21.113344 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:21.114758 31504 external_mini_cluster.cc:1428] Setting key 9ab029f6fe712e84e0da9f784c8736e8
I20260928 16:44:21.115729 31593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d: Bootstrap starting.
I20260928 16:44:21.118948 31593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:21.119952 31593 log.cc:824] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:21.122437 31593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d: No bootstrap required, opened a new log
I20260928 16:44:21.127926 31593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } }
I20260928 16:44:21.128268 31593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:21.128326 31593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e6eb0333fc4cff85b57c108a85350d, State: Initialized, Role: FOLLOWER
I20260928 16:44:21.128876 31593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } }
I20260928 16:44:21.129082 31593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:21.129150 31593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:21.129257 31593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:21.130348 31593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } }
I20260928 16:44:21.130760 31593 leader_election.cc:302] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e6eb0333fc4cff85b57c108a85350d; no voters:
I20260928 16:44:21.131151 31593 leader_election.cc:288] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:21.131397 31598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:21.131649 31598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [term 1 LEADER]: Becoming Leader. State: Replica: 27e6eb0333fc4cff85b57c108a85350d, State: Running, Role: LEADER
I20260928 16:44:21.132064 31598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } }
I20260928 16:44:21.132720 31593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:21.134052 31600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e6eb0333fc4cff85b57c108a85350d. Latest consensus state: current_term: 1 leader_uuid: "27e6eb0333fc4cff85b57c108a85350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } } }
I20260928 16:44:21.134056 31599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e6eb0333fc4cff85b57c108a85350d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e6eb0333fc4cff85b57c108a85350d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38301 } } }
I20260928 16:44:21.134303 31600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:21.134371 31599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:21.135077 31607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:21.139559 31607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:21.148324 31607 catalog_manager.cc:1347] Generated new cluster ID: e73dbf7b90ba469486a016a66cd8063c
I20260928 16:44:21.148411 31607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:21.148821 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.118305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.30.196.62:38301)
0928 16:44:21.118906 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.118928 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.120289 (+ 1361us) server_negotiation.cc:407] Connection header received
0928 16:44:21.125149 (+ 4860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.125169 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.125507 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.125808 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.129484 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.130602 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.132457 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.133244 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.133412 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.140418 (+ 7006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.140426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.140443 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.140483 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.143289 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.144395 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.144401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.144406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.144713 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.145256 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.145260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.145261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.145779 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.146061 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.146889 (+ 828us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.147286 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":396,"thread_start_us":175,"threads_started":1}
I20260928 16:44:21.164160 31607 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:21.165616 31607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:21.181154 31607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e6eb0333fc4cff85b57c108a85350d: Generated new TSK 0
I20260928 16:44:21.182391 31607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:21.226094 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.350749 31619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.351069 31619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.351150 31619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.355548 31619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.355676 31619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.355803 31619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:21.362522 31619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:21.363837 31619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.365144 31619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.375818 31624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.375876 31627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.375903 31625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.376334 31619 server_base.cc:1034] running on GCE node
I20260928 16:44:21.376765 31619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.377528 31619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.378805 31619 hybrid_clock.cc:648] HybridClock initialized: now 1790613861378754 us; error 62 us; skew 500 ppm
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.382129 31619 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:21.383175 31619 webserver.cc:460] Webserver started at http://127.30.196.1:40167/ using document root <none> and password file <none>
I20260928 16:44:21.383908 31619 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.384017 31619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.384261 31619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.386456 31619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "a3a46cb162884e0d89c7a03ef9a11780"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "81f28e81af882083f2a4fdd6645111bb"
server_key_iv: "e57a936b30b850ef2240a0db98d0c751"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.387148 31619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "a3a46cb162884e0d89c7a03ef9a11780"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "81f28e81af882083f2a4fdd6645111bb"
server_key_iv: "e57a936b30b850ef2240a0db98d0c751"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.391611 31619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 16:44:21.394665 31634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.396592 31619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:21.396765 31619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "a3a46cb162884e0d89c7a03ef9a11780"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "81f28e81af882083f2a4fdd6645111bb"
server_key_iv: "e57a936b30b850ef2240a0db98d0c751"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.396911 31619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.409888 31619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.414021 31619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.414259 31619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.414959 31619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:21.415992 31619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:21.416072 31619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.416167 31619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:21.416222 31619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.426412 31619 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:44769
I20260928 16:44:21.426433 31747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:44769 every 8 connection(s)
I20260928 16:44:21.427572 31619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:21.434350 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 31619
I20260928 16:44:21.434541 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:21.434808 31504 external_mini_cluster.cc:1428] Setting key abd8a4ab85a20aa9d88ed7fc4e7b3b91
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:21.444087 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.429667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:48767 (local address 127.30.196.62:38301)
0928 16:44:21.429856 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.429861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.430750 (+ 889us) server_negotiation.cc:407] Connection header received
0928 16:44:21.432333 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.432337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.432390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.432469 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.434102 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.434792 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.436170 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.436486 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.436607 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.439842 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.439849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.439855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.439891 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.441617 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.442243 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.442247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.442248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.442333 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.442736 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.442739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.442741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.442962 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.443084 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.443661 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.443880 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:21.445047 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.430087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38301 (local address 127.30.196.1:48767)
0928 16:44:21.430588 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:21.430632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:21.432095 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:21.432842 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:21.432849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:21.433251 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:21.433939 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.433955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.434961 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.434964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.436039 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.436049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.436652 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.436656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.436844 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.437730 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:21.437749 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:21.439571 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:21.441762 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.441770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.441792 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.442102 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.442448 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.442451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.442453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.442616 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.443085 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:21.443095 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:21.443378 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 16:44:21.443598 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 16:44:21.444068 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260928 16:44:21.446837 31748 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38301
I20260928 16:44:21.447136 31748 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:21.447829 31748 heartbeater.cc:511] Master 127.30.196.62:38301 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:21.450404 31556 ts_manager.cc:194] Registered new tserver with Master: a3a46cb162884e0d89c7a03ef9a11780 (127.30.196.1:44769)
I20260928 16:44:21.451869 31556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:48767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:21.468528 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.587567 31753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.587970 31753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.588076 31753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.592254 31753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.592346 31753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.592437 31753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:21.598129 31753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:21.599458 31753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.600797 31753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.611886 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.611837 31762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.612056 31753 server_base.cc:1034] running on GCE node
W20260928 16:44:21.611824 31760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.612782 31753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.613548 31753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.614928 31753 hybrid_clock.cc:648] HybridClock initialized: now 1790613861614900 us; error 34 us; skew 500 ppm
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.618739 31753 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:21.619939 31753 webserver.cc:460] Webserver started at http://127.30.196.2:40463/ using document root <none> and password file <none>
I20260928 16:44:21.620564 31753 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.620635 31753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.620828 31753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.623214 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "b13228740a404145a9e90e5c79ec70d7"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "4bd6ad2faf0b7783a66d2fe786f0a82d"
server_key_iv: "e5bc7ab5a1f14ade4d8fdacd59d5f62c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.623929 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "b13228740a404145a9e90e5c79ec70d7"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "4bd6ad2faf0b7783a66d2fe786f0a82d"
server_key_iv: "e5bc7ab5a1f14ade4d8fdacd59d5f62c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.630075 31753 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260928 16:44:21.634368 31769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.636631 31753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:21.636791 31753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "b13228740a404145a9e90e5c79ec70d7"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "4bd6ad2faf0b7783a66d2fe786f0a82d"
server_key_iv: "e5bc7ab5a1f14ade4d8fdacd59d5f62c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.636932 31753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.655611 31753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.660671 31753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.660880 31753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.661530 31753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:21.662468 31753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:21.662523 31753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.662569 31753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:21.662586 31753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.673777 31753 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:42845
I20260928 16:44:21.673846 31882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:42845 every 8 connection(s)
I20260928 16:44:21.674886 31753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:21.676756 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 31753
I20260928 16:44:21.676887 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:21.677168 31504 external_mini_cluster.cc:1428] Setting key 61fc870585215da98c4705cdacda8207
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:21.694007 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.676996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:57497 (local address 127.30.196.62:38301)
0928 16:44:21.677159 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.677164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.678219 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 16:44:21.680200 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.680210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.680258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.680344 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.681750 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.682380 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.684425 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.684666 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.684761 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.688174 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.688178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.688181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.688218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.690349 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.691493 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.691497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.691498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.691602 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.692269 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.692272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.692273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.692487 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.692640 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.693742 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.693875 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:21.695518 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.677501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38301 (local address 127.30.196.2:57497)
0928 16:44:21.678075 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:21.678120 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:21.679922 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:21.680586 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:21.680600 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:21.681045 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:21.681597 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.681611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.682547 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.682553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.684266 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.684283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.684854 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.684861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.685122 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.685871 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:21.685895 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:21.687946 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:21.690655 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.690665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.690687 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.691299 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.691846 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.691855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.691861 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.692125 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.692765 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:21.692776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:21.693288 (+ 512us) client_negotiation.cc:769] Sending connection context
0928 16:44:21.693612 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 16:44:21.694296 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":157,"threads_started":1}
I20260928 16:44:21.696907 31883 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38301
I20260928 16:44:21.697189 31883 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:21.697757 31883 heartbeater.cc:511] Master 127.30.196.62:38301 requested a full tablet report, sending...
I20260928 16:44:21.698944 31556 ts_manager.cc:194] Registered new tserver with Master: b13228740a404145a9e90e5c79ec70d7 (127.30.196.2:42845)
I20260928 16:44:21.699978 31556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:57497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:21.716847 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.838804 31888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.839185 31888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.839268 31888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.843781 31888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.843879 31888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.843974 31888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:21.849704 31888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:21.850871 31888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.851995 31888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.863044 31893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.863062 31896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.863101 31894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.864042 31888 server_base.cc:1034] running on GCE node
I20260928 16:44:21.864596 31888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.865378 31888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.866593 31888 hybrid_clock.cc:648] HybridClock initialized: now 1790613861866570 us; error 38 us; skew 500 ppm
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.869997 31888 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:21.871125 31888 webserver.cc:460] Webserver started at http://127.30.196.3:35113/ using document root <none> and password file <none>
I20260928 16:44:21.871800 31888 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.871896 31888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.872143 31888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.874358 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "a1249426a24c431886efe007d2a92576"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "0ba3b6f6f330b3bf2c001a726293e805"
server_key_iv: "2cefbeb0a348ff9870cf2303d1438913"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.874974 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "a1249426a24c431886efe007d2a92576"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "0ba3b6f6f330b3bf2c001a726293e805"
server_key_iv: "2cefbeb0a348ff9870cf2303d1438913"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.879062 31888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:21.881851 31903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.883286 31888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:21.883450 31888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "a1249426a24c431886efe007d2a92576"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-5rg1"
server_key: "0ba3b6f6f330b3bf2c001a726293e805"
server_key_iv: "2cefbeb0a348ff9870cf2303d1438913"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.883582 31888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.907052 31888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.910805 31888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.911057 31888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.911774 31888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:21.912671 31888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:21.912747 31888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.912817 31888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:21.912860 31888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.923383 31888 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:40213
I20260928 16:44:21.923585 32016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:40213 every 8 connection(s)
I20260928 16:44:21.924466 31888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:21.924875 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 31888
I20260928 16:44:21.925031 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:21.925355 31504 external_mini_cluster.cc:1428] Setting key 21899cdcd91a9995062a305848b9c22f
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:21.940752 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.926675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:33527 (local address 127.30.196.62:38301)
0928 16:44:21.926901 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.926908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.928305 (+ 1397us) server_negotiation.cc:407] Connection header received
0928 16:44:21.929347 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.929350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.929408 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.929494 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.930695 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.931246 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.932496 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.932731 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.932847 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.935850 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.935853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.935855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.935880 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.937482 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.938267 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.938270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.938271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.938372 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.938813 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.938816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.938817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.939041 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.939174 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.940020 (+ 846us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.940291 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:44:21.941386 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.927076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38301 (local address 127.30.196.3:33527)
0928 16:44:21.928141 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:21.928202 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:21.929143 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:21.929673 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:21.929681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:21.930062 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:21.930551 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.930565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.931375 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.931379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.932362 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.932372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.932909 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.932914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.933098 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.933731 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:21.933757 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:21.935632 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:21.937669 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.937674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.937689 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.937990 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.938499 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.938502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.938503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.938666 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.939226 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:21.939234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:21.939525 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 16:44:21.939785 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 16:44:21.940226 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":164,"threads_started":1}
I20260928 16:44:21.942596 32017 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38301
I20260928 16:44:21.942965 32017 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:21.943527 32017 heartbeater.cc:511] Master 127.30.196.62:38301 requested a full tablet report, sending...
I20260928 16:44:21.944729 31556 ts_manager.cc:194] Registered new tserver with Master: a1249426a24c431886efe007d2a92576 (127.30.196.3:40213)
I20260928 16:44:21.945569 31556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:33527
I20260928 16:44:21.954527 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:21.983788 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.973059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.30.196.1:44769)
0928 16:44:21.973468 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.973475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.973500 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:44:21.973584 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.973590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.973792 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.973922 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.974908 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.975801 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.976717 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.977115 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.977211 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.979855 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.979866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.979881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.979908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.981611 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.982141 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.982147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.982148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.982244 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.982553 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.982556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.982557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.982818 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.982982 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.983413 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.983556 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":170,"threads_started":1}
W20260928 16:44:21.984886 31662 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896
Sep 28 16:44:21 dist-test-slave-5rg1 krb5kdc[31511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:44:22 dist-test-slave-5rg1 krb5kdc[31511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:22.024675 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:22.012599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.30.196.1:44769)
0928 16:44:22.012767 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:22.012774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:22.012794 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:22.012975 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:22.012979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:22.013045 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:22.013206 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:22.014010 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.014558 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.016066 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.016307 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.016431 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:22.020067 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:22.020070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:22.020073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:22.020107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:22.022084 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.022639 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.022643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.022645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.022738 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.023613 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.023619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.023620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.023902 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:22.024069 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:22.024334 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 16:44:22.024490 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:22.026142 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 31619
I20260928 16:44:22.033033 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 31753
I20260928 16:44:22.039258 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 31888
I20260928 16:44:22.045042 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 31524
2026-09-28T16:44:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2430 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:22 dist-test-slave-5rg1 krb5kdc[32031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:22 dist-test-slave-5rg1 krb5kdc[32031](info): set up 2 sockets
Sep 28 16:44:22 dist-test-slave-5rg1 krb5kdc[32031](info): commencing operation
krb5kdc: starting...
W20260928 16:44:23.148026 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:23 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:23.299499 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33263
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:33263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:23.427178 32044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:23.427532 32044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:23.427628 32044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:23.431939 32044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:23.432058 32044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:23.432096 32044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:23.432134 32044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:23.432168 32044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:23.437987 32044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:33263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:23.439260 32044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:23.440404 32044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:23.450888 32050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:23.451102 32044 server_base.cc:1034] running on GCE node
W20260928 16:44:23.450958 32052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:23.450910 32049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:23.452157 32044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:23.453467 32044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:23.456844 32044 hybrid_clock.cc:648] HybridClock initialized: now 1790613863456771 us; error 59 us; skew 500 ppm
Sep 28 16:44:23 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:23.470458 32044 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:23.471575 32044 webserver.cc:460] Webserver started at http://127.30.196.62:45475/ using document root <none> and password file <none>
I20260928 16:44:23.472294 32044 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:23.472369 32044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:23.472600 32044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:23.474843 32044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "196f8c821a174056be478c58ab0c73d7"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "f5855ea847d0fbabe32559192e2770b3"
server_key_iv: "b89f117037ceca2e6fa1fafcb5d8171e"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.475452 32044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "196f8c821a174056be478c58ab0c73d7"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "f5855ea847d0fbabe32559192e2770b3"
server_key_iv: "b89f117037ceca2e6fa1fafcb5d8171e"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.480137 32044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:23.483608 32059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:23.485193 32044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:23.485363 32044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "196f8c821a174056be478c58ab0c73d7"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "f5855ea847d0fbabe32559192e2770b3"
server_key_iv: "b89f117037ceca2e6fa1fafcb5d8171e"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.485504 32044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:23.498329 32044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:23.501958 32044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:23.502144 32044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:23.511835 32044 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:33263
I20260928 16:44:23.511984 32111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:33263 every 8 connection(s)
I20260928 16:44:23.512979 32044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:23.516172 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:23.517889 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32044
I20260928 16:44:23.517989 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:23.518275 31504 external_mini_cluster.cc:1428] Setting key dfaf74826dfad181c90f7333040d5a99
I20260928 16:44:23.524271 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7: Bootstrap starting.
I20260928 16:44:23.526885 32112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:23.527786 32112 log.cc:824] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:44:23 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:23.530310 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7: No bootstrap required, opened a new log
I20260928 16:44:23.534827 32112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } }
I20260928 16:44:23.535082 32112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:23.535118 32112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196f8c821a174056be478c58ab0c73d7, State: Initialized, Role: FOLLOWER
I20260928 16:44:23.535620 32112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } }
I20260928 16:44:23.535818 32112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:23.535904 32112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:23.536008 32112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:23.536015 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.520297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.30.196.62:33263)
0928 16:44:23.520854 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:23.520870 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:23.520922 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:44:23.521676 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:23.521700 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:23.522088 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:23.522531 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:23.523458 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.524666 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.525579 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.526106 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.526294 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.529556 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:23.529563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:23.529586 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:23.529647 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:23.532236 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.532851 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.532857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.532862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.533026 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.533346 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.533349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.533350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.533774 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:23.533941 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:23.534338 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 16:44:23.534768 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":169,"threads_started":1}
I20260928 16:44:23.537065 32112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } }
I20260928 16:44:23.537456 32112 leader_election.cc:302] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 196f8c821a174056be478c58ab0c73d7; no voters:
I20260928 16:44:23.537853 32112 leader_election.cc:288] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:23.537997 32117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:23.538283 32117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [term 1 LEADER]: Becoming Leader. State: Replica: 196f8c821a174056be478c58ab0c73d7, State: Running, Role: LEADER
I20260928 16:44:23.538628 32117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } }
I20260928 16:44:23.539152 32112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:23.539956 32119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 196f8c821a174056be478c58ab0c73d7. Latest consensus state: current_term: 1 leader_uuid: "196f8c821a174056be478c58ab0c73d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } } }
I20260928 16:44:23.540211 32119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:23.540416 32118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "196f8c821a174056be478c58ab0c73d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196f8c821a174056be478c58ab0c73d7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33263 } } }
I20260928 16:44:23.540517 32118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:23.540680 32126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:23.543466 32126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:23.548862 32126 catalog_manager.cc:1347] Generated new cluster ID: aedd7ffa1edb44feb0211e90a74e7ad8
I20260928 16:44:23.548960 32126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:23.571702 32126 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:23.572784 32126 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:23.577618 32126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 196f8c821a174056be478c58ab0c73d7: Generated new TSK 0
I20260928 16:44:23.578994 32126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:23.623823 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:23.749428 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:23.749871 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:23.749994 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:23.754393 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:23.754506 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:23.754621 32139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:23.762348 32139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:23.763696 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:23.764817 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:23.775542 32145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:23.775418 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:23.775419 32144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:23.775575 32139 server_base.cc:1034] running on GCE node
I20260928 16:44:23.776449 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:23.777212 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:23.778432 32139 hybrid_clock.cc:648] HybridClock initialized: now 1790613863778409 us; error 36 us; skew 500 ppm
Sep 28 16:44:23 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:23.781581 32139 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:23.782507 32139 webserver.cc:460] Webserver started at http://127.30.196.1:44105/ using document root <none> and password file <none>
I20260928 16:44:23.783087 32139 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:23.783152 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:23.783325 32139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:23.785751 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "afe67834a3a048a2b67db9a829cc2a8b"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "17a69293c61ed06ee02ec7069c41fe07"
server_key_iv: "74918fee462e2737a7fbbd815473705b"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.786330 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "afe67834a3a048a2b67db9a829cc2a8b"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "17a69293c61ed06ee02ec7069c41fe07"
server_key_iv: "74918fee462e2737a7fbbd815473705b"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.791224 32139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 16:44:23.794657 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:23.796583 32139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:23.796720 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "afe67834a3a048a2b67db9a829cc2a8b"
format_stamp: "Formatted at 2026-09-28 16:44:23 on dist-test-slave-5rg1"
server_key: "17a69293c61ed06ee02ec7069c41fe07"
server_key_iv: "74918fee462e2737a7fbbd815473705b"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.796838 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:23.818388 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:23.822705 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:23.822893 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:23.823563 32139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:23.824575 32139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:23.824630 32139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:23.824674 32139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:23.824690 32139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:23.835114 32139 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:43521
I20260928 16:44:23.835135 32269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:43521 every 8 connection(s)
I20260928 16:44:23.836143 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:23.842562 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32139
I20260928 16:44:23.842666 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:23.842929 31504 external_mini_cluster.cc:1428] Setting key 3d8cb8b9ec34fa44ca04ed2cb66bd42d
Sep 28 16:44:23 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:23.853768 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.838245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:60829 (local address 127.30.196.62:33263)
0928 16:44:23.838410 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:23.838416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:23.839533 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 16:44:23.840733 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:23.840737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:23.840790 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:23.840878 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:23.842305 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.842947 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.844582 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.844902 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.845022 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.848148 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:23.848151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:23.848153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:23.848179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:23.850166 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.851104 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.851108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.851109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.851206 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.851883 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.851886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.851887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.852223 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:23.852317 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:23.853264 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 16:44:23.853530 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:23.854486 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.838733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33263 (local address 127.30.196.1:60829)
0928 16:44:23.839295 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:23.839341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:23.840469 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:23.841047 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:23.841054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:23.841598 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:23.842108 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:23.842128 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.843099 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.843103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:23.844445 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:23.844458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.845108 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.845112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:23.845275 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.845949 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:23.845967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:23.847931 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:23.850478 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:23.850490 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:23.850513 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:23.850955 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:23.851315 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:23.851320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:23.851324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:23.851590 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:23.852356 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:23.852363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:23.852764 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 16:44:23.853034 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:44:23.853425 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260928 16:44:23.856230 32270 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33263
I20260928 16:44:23.856511 32270 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:23.857261 32270 heartbeater.cc:511] Master 127.30.196.62:33263 requested a full tablet report, sending...
I20260928 16:44:23.859232 32076 ts_manager.cc:194] Registered new tserver with Master: afe67834a3a048a2b67db9a829cc2a8b (127.30.196.1:43521)
I20260928 16:44:23.860697 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:60829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:23.876575 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:24.003295 32275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:24.003623 32275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:24.003767 32275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:24.010455 32275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:24.010596 32275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:24.010675 32275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:24.019061 32275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:24.020418 32275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:24.021586 32275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:24.032104 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.032029 32281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.032040 32283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:24.032656 32275 server_base.cc:1034] running on GCE node
I20260928 16:44:24.033111 32275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:24.033835 32275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:24.037209 32275 hybrid_clock.cc:648] HybridClock initialized: now 1790613864037171 us; error 137 us; skew 500 ppm
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:24.041649 32275 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:24.042994 32275 webserver.cc:460] Webserver started at http://127.30.196.2:37997/ using document root <none> and password file <none>
I20260928 16:44:24.044134 32275 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:24.044245 32275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:24.044551 32275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:24.047266 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "eaa9a1f081594a95a1eb9e9dfa5861db"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "fdb8533b11980e0795625840ff117fb8"
server_key_iv: "26f3e3f12a2ac5ad348ab8115f1d588d"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.047912 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "eaa9a1f081594a95a1eb9e9dfa5861db"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "fdb8533b11980e0795625840ff117fb8"
server_key_iv: "26f3e3f12a2ac5ad348ab8115f1d588d"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.052127 32275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:24.055550 32290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.057096 32275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:24.057224 32275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "eaa9a1f081594a95a1eb9e9dfa5861db"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "fdb8533b11980e0795625840ff117fb8"
server_key_iv: "26f3e3f12a2ac5ad348ab8115f1d588d"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.057322 32275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:24.093199 32275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:24.097379 32275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:24.097564 32275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:24.098121 32275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:24.099148 32275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:24.099224 32275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.099328 32275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:24.099373 32275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.111889 32275 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:40827
I20260928 16:44:24.111915 32404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:40827 every 8 connection(s)
I20260928 16:44:24.112906 32275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:24.116183 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32275
I20260928 16:44:24.116325 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:24.116670 31504 external_mini_cluster.cc:1428] Setting key d79279113bb2242dbf48726ad53b5592
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:24.136489 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.115757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:40975 (local address 127.30.196.62:33263)
0928 16:44:24.115912 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.115915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.117950 (+ 2035us) server_negotiation.cc:407] Connection header received
0928 16:44:24.119835 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.119840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.119934 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.120053 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.122068 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.122718 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.124568 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.124837 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.124984 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.132135 (+ 7151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.132139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.132142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.132175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.133830 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.134503 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.134507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.134509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.134591 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.134989 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.134993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.134994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.135229 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.135349 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.136066 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.136229 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:44:24.137418 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.116573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33263 (local address 127.30.196.2:40975)
0928 16:44:24.117746 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.117793 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.119184 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.120411 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.120422 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.121008 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:24.121799 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.121823 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.122959 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.122967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.124369 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.124388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.125069 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.125074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.125314 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.126312 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:24.126334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:24.131851 (+ 5517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:24.133998 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:24.134005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:24.134034 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:24.134364 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:24.134703 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:24.134706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:24.134708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:24.134873 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:24.135364 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:24.135374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:24.135777 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.135993 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.136439 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":199,"threads_started":1}
I20260928 16:44:24.138820 32405 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33263
I20260928 16:44:24.139120 32405 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:24.139846 32405 heartbeater.cc:511] Master 127.30.196.62:33263 requested a full tablet report, sending...
I20260928 16:44:24.141299 32076 ts_manager.cc:194] Registered new tserver with Master: eaa9a1f081594a95a1eb9e9dfa5861db (127.30.196.2:40827)
I20260928 16:44:24.142014 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:40975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:24.147603 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:24.274217 32410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:24.274500 32410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:24.274549 32410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:24.278805 32410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:24.278889 32410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:24.278954 32410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:24.284832 32410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33263
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:24.286231 32410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:24.287493 32410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:24.298712 32418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.298763 32415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:24.298867 32410 server_base.cc:1034] running on GCE node
W20260928 16:44:24.298712 32416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:24.300055 32410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:24.300983 32410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:24.302260 32410 hybrid_clock.cc:648] HybridClock initialized: now 1790613864302243 us; error 48 us; skew 500 ppm
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:24.306435 32410 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:24.307929 32410 webserver.cc:460] Webserver started at http://127.30.196.3:37197/ using document root <none> and password file <none>
I20260928 16:44:24.308817 32410 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:24.308934 32410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:24.309201 32410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:24.312410 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "ca87ffd46b834ce8a9f56b2e4888d091"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "cf5f065c72d2b899928685653358136d"
server_key_iv: "4d11047d5b0c628e8fe112ae477c5b9b"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.313299 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "ca87ffd46b834ce8a9f56b2e4888d091"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "cf5f065c72d2b899928685653358136d"
server_key_iv: "4d11047d5b0c628e8fe112ae477c5b9b"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.319962 32410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:44:24.323566 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.325349 32410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:24.325496 32410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "ca87ffd46b834ce8a9f56b2e4888d091"
format_stamp: "Formatted at 2026-09-28 16:44:24 on dist-test-slave-5rg1"
server_key: "cf5f065c72d2b899928685653358136d"
server_key_iv: "4d11047d5b0c628e8fe112ae477c5b9b"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.325629 32410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:24.337491 32410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:24.341931 32410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:24.342119 32410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:24.342698 32410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:24.343568 32410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:24.343624 32410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.343703 32410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:24.343775 32410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.353937 32410 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:40215
I20260928 16:44:24.354094 32539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:40215 every 8 connection(s)
I20260928 16:44:24.354861 32410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:24.355222 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32410
I20260928 16:44:24.355372 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:24.355765 31504 external_mini_cluster.cc:1428] Setting key e5752c7658f892b3b8acaf4f19723947
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:24.369879 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.357054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:45461 (local address 127.30.196.62:33263)
0928 16:44:24.357273 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.357279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.358538 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 16:44:24.359548 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.359551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.359611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.359750 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.360999 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.361578 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.362798 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.363035 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.363160 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.365990 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.365996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.366003 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.366028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.367573 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.368159 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.368162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.368164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.368247 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.368605 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.368610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.368612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.368887 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.369025 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.369563 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.369702 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:44:24.370956 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.357724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33263 (local address 127.30.196.3:45461)
0928 16:44:24.358390 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.358442 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.359282 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.359929 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.359937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.360308 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:24.360812 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.360825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.361710 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.361719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.362661 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.362672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.363205 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.363209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.363388 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.364044 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:24.364061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:24.365801 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:24.367736 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:24.367744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:24.367760 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:24.368043 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:24.368338 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:24.368341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:24.368343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:24.368505 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:24.369015 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:24.369022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:24.369310 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.369524 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.369901 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":188,"threads_started":1}
I20260928 16:44:24.372257 32540 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33263
I20260928 16:44:24.372538 32540 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:24.373064 32540 heartbeater.cc:511] Master 127.30.196.62:33263 requested a full tablet report, sending...
I20260928 16:44:24.374327 32076 ts_manager.cc:194] Registered new tserver with Master: ca87ffd46b834ce8a9f56b2e4888d091 (127.30.196.3:40215)
I20260928 16:44:24.374974 32076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:45461
I20260928 16:44:24.384114 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:24.397897 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.389708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.30.196.62:33263)
0928 16:44:24.389844 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.389848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.390163 (+ 315us) server_negotiation.cc:407] Connection header received
0928 16:44:24.390286 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.390288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.390334 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.390485 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.391227 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.391845 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.392671 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.392875 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.392965 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.394420 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.394423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.394425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.394447 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.396196 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.396625 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.396628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.396629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.396705 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.397014 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.397016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.397018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.397261 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.397384 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.397653 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.397757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:44:24.403851 32076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:24.407038 32076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:24.425457 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.419231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.30.196.3:40215)
0928 16:44:24.419707 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.419724 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.419748 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:44:24.419843 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.419855 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.420054 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.420369 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:24.420585 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.421783 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.423887 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.424752 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.424891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.425039 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.425170 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.425281 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":124,"threads_started":1}
I20260928 16:44:24.425457 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.418692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:40215 (local address 127.0.0.1:51722)
0928 16:44:24.419273 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.419330 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.419468 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.420243 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.420250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.420291 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:24.420460 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.420467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.422135 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.422142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.423653 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.423704 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.424906 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.424911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.425015 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.425042 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.425140 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.425295 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":17,"spinlock_wait_cycles":7296,"thread_start_us":119,"threads_started":1}
I20260928 16:44:24.425474 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.418697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.30.196.1:43521)
0928 16:44:24.419009 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.419016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.419422 (+ 406us) server_negotiation.cc:407] Connection header received
0928 16:44:24.419649 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.419655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.419907 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.420038 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:24.420798 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.421782 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.423452 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.424349 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.424472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.424639 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.425055 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.425243 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260928 16:44:24.426040 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.418505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:43521 (local address 127.0.0.1:38542)
0928 16:44:24.419273 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.419330 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.419478 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.420250 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.420255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.420291 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:24.420662 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.420669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.421930 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.421933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.423305 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.423314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.424515 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.424519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.424677 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.424738 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.425669 (+ 931us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.425776 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":17,"spinlock_wait_cycles":5504,"thread_start_us":120,"threads_started":1}
I20260928 16:44:24.426431 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.419749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.30.196.2:40827)
0928 16:44:24.420199 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.420212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.420233 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:24.420488 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.420495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.420697 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.420837 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:24.421272 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.422331 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.423967 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.424940 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.425061 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.425250 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.426090 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.426254 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":148,"threads_started":1}
I20260928 16:44:24.426234 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.419310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:40827 (local address 127.0.0.1:50496)
0928 16:44:24.419789 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.419802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.420326 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.421000 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.421003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.421021 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:24.421128 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.421133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.422529 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.422533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.423815 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.423836 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.425332 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.425336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.425423 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.425445 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.425527 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.426089 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
I20260928 16:44:24.429483 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 58813d12789e430d85ced3c1d4a61940 (DEFAULT_TABLE table=test-table [id=395305b58263403c97a4dada1bbe3207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:24.429482 32339 tablet_service.cc:1467] Processing CreateTablet for tablet 58813d12789e430d85ced3c1d4a61940 (DEFAULT_TABLE table=test-table [id=395305b58263403c97a4dada1bbe3207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:24.429768 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 58813d12789e430d85ced3c1d4a61940 (DEFAULT_TABLE table=test-table [id=395305b58263403c97a4dada1bbe3207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:24.430692 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58813d12789e430d85ced3c1d4a61940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:24.430692 32339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58813d12789e430d85ced3c1d4a61940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:24.430938 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58813d12789e430d85ced3c1d4a61940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:24.437816 32557 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Bootstrap starting.
I20260928 16:44:24.440268 32559 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Bootstrap starting.
I20260928 16:44:24.440461 32558 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Bootstrap starting.
I20260928 16:44:24.441419 32557 tablet_bootstrap.cc:654] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:24.442394 32557 log.cc:824] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:24.442922 32559 tablet_bootstrap.cc:654] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:24.443408 32558 tablet_bootstrap.cc:654] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:24.444279 32558 log.cc:824] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:24.444679 32559 log.cc:824] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:24.445473 32557 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: No bootstrap required, opened a new log
I20260928 16:44:24.445745 32557 ts_tablet_manager.cc:1397] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:44:24.446236 32558 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: No bootstrap required, opened a new log
I20260928 16:44:24.446460 32558 ts_tablet_manager.cc:1397] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 16:44:24.450253 32558 raft_consensus.cc:348] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.450569 32558 raft_consensus.cc:374] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:24.450666 32558 raft_consensus.cc:729] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa9a1f081594a95a1eb9e9dfa5861db, State: Initialized, Role: FOLLOWER
I20260928 16:44:24.450842 32557 raft_consensus.cc:348] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.451189 32557 raft_consensus.cc:374] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:24.451262 32557 raft_consensus.cc:729] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe67834a3a048a2b67db9a829cc2a8b, State: Initialized, Role: FOLLOWER
I20260928 16:44:24.451185 32558 consensus_queue.cc:260] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.451728 32557 consensus_queue.cc:260] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.452351 32558 ts_tablet_manager.cc:1428] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 16:44:24.452453 32270 heartbeater.cc:503] Master 127.30.196.62:33263 was elected leader, sending a full tablet report...
I20260928 16:44:24.452708 32557 ts_tablet_manager.cc:1428] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:44:24.452877 32559 tablet_bootstrap.cc:492] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: No bootstrap required, opened a new log
I20260928 16:44:24.453085 32559 ts_tablet_manager.cc:1397] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260928 16:44:24.453213 32405 heartbeater.cc:503] Master 127.30.196.62:33263 was elected leader, sending a full tablet report...
I20260928 16:44:24.457017 32559 raft_consensus.cc:348] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.457315 32559 raft_consensus.cc:374] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:24.457391 32559 raft_consensus.cc:729] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca87ffd46b834ce8a9f56b2e4888d091, State: Initialized, Role: FOLLOWER
I20260928 16:44:24.457896 32559 consensus_queue.cc:260] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.458727 32540 heartbeater.cc:503] Master 127.30.196.62:33263 was elected leader, sending a full tablet report...
I20260928 16:44:24.458922 32559 ts_tablet_manager.cc:1428] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:44:24.534730 32564 raft_consensus.cc:482] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:24.535001 32564 raft_consensus.cc:504] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.536351 32564 leader_election.cc:288] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca87ffd46b834ce8a9f56b2e4888d091 (127.30.196.3:40215), eaa9a1f081594a95a1eb9e9dfa5861db (127.30.196.2:40827)
I20260928 16:44:24.542003 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.537371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:50131 (local address 127.30.196.2:40827)
0928 16:44:24.537507 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.537511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.537668 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:44:24.537756 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.537758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.537816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.537902 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:24.538223 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.538949 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.540646 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.541314 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.541450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.541509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.541785 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.541863 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:44:24.542003 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.536830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:40093 (local address 127.30.196.3:40215)
0928 16:44:24.536966 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.536969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.537139 (+ 170us) server_negotiation.cc:407] Connection header received
0928 16:44:24.537316 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.537319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.537374 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.537466 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:24.537857 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.538753 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.540632 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.541319 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.541450 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.541509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.541785 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.541863 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:24.542194 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.536710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:40215 (local address 127.30.196.1:40093)
0928 16:44:24.537037 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.537053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.537184 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.537549 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.537552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.537585 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:24.537744 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.537753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.538871 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.538874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.540505 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.540512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.541497 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.541501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.541610 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.541652 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.541742 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.541960 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":154,"thread_start_us":147,"threads_started":1}
I20260928 16:44:24.542338 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.537246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:40827 (local address 127.30.196.1:50131)
0928 16:44:24.537562 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.537576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.537669 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.537949 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.537952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.537972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:24.538104 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.538110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.539046 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.539049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.540492 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.540505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.541547 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.541550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.541642 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.541658 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:44:24.541742 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:44:24.541960 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":127,"thread_start_us":118,"threads_started":1}
I20260928 16:44:24.542918 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58813d12789e430d85ced3c1d4a61940" candidate_uuid: "afe67834a3a048a2b67db9a829cc2a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" is_pre_election: true
I20260928 16:44:24.542908 32494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58813d12789e430d85ced3c1d4a61940" candidate_uuid: "afe67834a3a048a2b67db9a829cc2a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" is_pre_election: true
I20260928 16:44:24.543241 32359 raft_consensus.cc:2454] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe67834a3a048a2b67db9a829cc2a8b in term 0.
I20260928 16:44:24.543244 32494 raft_consensus.cc:2454] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe67834a3a048a2b67db9a829cc2a8b in term 0.
I20260928 16:44:24.543792 32158 leader_election.cc:302] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe67834a3a048a2b67db9a829cc2a8b, ca87ffd46b834ce8a9f56b2e4888d091; no voters:
I20260928 16:44:24.544111 32564 raft_consensus.cc:2781] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:24.544210 32564 raft_consensus.cc:482] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:24.544245 32564 raft_consensus.cc:3037] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:24.545382 32564 raft_consensus.cc:504] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.545789 32564 leader_election.cc:288] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [CANDIDATE]: Term 1 election: Requested vote from peers ca87ffd46b834ce8a9f56b2e4888d091 (127.30.196.3:40215), eaa9a1f081594a95a1eb9e9dfa5861db (127.30.196.2:40827)
I20260928 16:44:24.546393 32494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58813d12789e430d85ced3c1d4a61940" candidate_uuid: "afe67834a3a048a2b67db9a829cc2a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca87ffd46b834ce8a9f56b2e4888d091"
I20260928 16:44:24.546497 32494 raft_consensus.cc:3037] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:24.546657 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58813d12789e430d85ced3c1d4a61940" candidate_uuid: "afe67834a3a048a2b67db9a829cc2a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db"
I20260928 16:44:24.546770 32359 raft_consensus.cc:3037] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:24.547610 32494 raft_consensus.cc:2454] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe67834a3a048a2b67db9a829cc2a8b in term 1.
I20260928 16:44:24.548008 32158 leader_election.cc:302] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe67834a3a048a2b67db9a829cc2a8b, ca87ffd46b834ce8a9f56b2e4888d091; no voters:
I20260928 16:44:24.548038 32359 raft_consensus.cc:2454] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe67834a3a048a2b67db9a829cc2a8b in term 1.
I20260928 16:44:24.548207 32564 raft_consensus.cc:2781] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:24.548440 32564 raft_consensus.cc:686] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [term 1 LEADER]: Becoming Leader. State: Replica: afe67834a3a048a2b67db9a829cc2a8b, State: Running, Role: LEADER
I20260928 16:44:24.548838 32564 consensus_queue.cc:237] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } }
I20260928 16:44:24.552199 32076 catalog_manager.cc:5495] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b reported cstate change: term changed from 0 to 1, leader changed from <none> to afe67834a3a048a2b67db9a829cc2a8b (127.30.196.1). New cstate: current_term: 1 leader_uuid: "afe67834a3a048a2b67db9a829cc2a8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe67834a3a048a2b67db9a829cc2a8b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 43521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca87ffd46b834ce8a9f56b2e4888d091" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaa9a1f081594a95a1eb9e9dfa5861db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40827 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
W20260928 16:44:24.587991 32271 tablet.cc:2369] T 58813d12789e430d85ced3c1d4a61940 P afe67834a3a048a2b67db9a829cc2a8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:24.590729 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.580374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.30.196.1:43521)
0928 16:44:24.580553 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.580559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.580714 (+ 155us) server_negotiation.cc:407] Connection header received
0928 16:44:24.580888 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.580891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.581014 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.581112 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.581817 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.582335 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.583206 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.583427 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.583539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.586494 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.586508 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.586527 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.586555 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.588667 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.589188 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.589194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.589195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.589279 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.589634 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.589640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.589645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.589920 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.590171 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.590383 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.590517 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:44:24.606376 32541 tablet.cc:2369] T 58813d12789e430d85ced3c1d4a61940 P ca87ffd46b834ce8a9f56b2e4888d091: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 16:44:24.614873 32406 tablet.cc:2369] T 58813d12789e430d85ced3c1d4a61940 P eaa9a1f081594a95a1eb9e9dfa5861db: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:24.624950 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.613930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.30.196.1:43521)
0928 16:44:24.614175 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.614181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.614288 (+ 107us) server_negotiation.cc:407] Connection header received
0928 16:44:24.614482 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.614485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.614536 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.614637 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.615438 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.616088 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.616958 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.617210 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.617340 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.620598 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.620602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.620605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.620654 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.622468 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.623378 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.623384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.623388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.623574 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.623940 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.623944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.623946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.624145 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.624298 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.624629 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.624772 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:24.656313 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.646422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.30.196.1:43521)
0928 16:44:24.646645 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.646652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.646984 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:44:24.647194 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.647197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.647244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.647327 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.648232 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.648747 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.649705 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.649932 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.650018 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.652896 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:24.652899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:24.652901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:24.652929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:24.654418 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.654872 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.654875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.654877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.654953 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:24.655270 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:24.655273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:24.655275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:24.655459 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:24.655604 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:24.655913 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:44:24.656119 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260928 16:44:24.657729 32183 tablet_service.cc:2908] Rejecting scan request for tablet 58813d12789e430d85ced3c1d4a61940: Uninitialized: clean time has not yet been initialized
I20260928 16:44:24.660421 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32139
I20260928 16:44:24.667459 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32275
I20260928 16:44:24.673800 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32410
I20260928 16:44:24.680186 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32044
2026-09-28T16:44:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2634 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32582](info): set up 2 sockets
Sep 28 16:44:24 dist-test-slave-5rg1 krb5kdc[32582](info): commencing operation
krb5kdc: starting...
W20260928 16:44:25.748337 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:25 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:25.888451 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:40063
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:40063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:26.005755 32595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:26.006233 32595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:26.006354 32595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:26.011075 32595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:26.011212 32595 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:44:26.011245 32595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:26.011291 32595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:26.011354 32595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:26.011394 32595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:26.017863 32595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:40063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:40063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:26.019181 32595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:26.020355 32595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:26.030512 32600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.030539 32601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.030622 32603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:26.030561 32595 server_base.cc:1034] running on GCE node
I20260928 16:44:26.031415 32595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:26.032626 32595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:26.033844 32595 hybrid_clock.cc:648] HybridClock initialized: now 1790613866033822 us; error 36 us; skew 500 ppm
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:26.037204 32595 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:26.038257 32595 webserver.cc:460] Webserver started at http://127.30.196.62:41083/ using document root <none> and password file <none>
I20260928 16:44:26.038861 32595 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:26.038959 32595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:26.039201 32595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:26.041574 32595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "87e35e733b7d487ab250392f019874eb"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5cf9cb630d2b13ce87d7df2b53e9ccd5"
server_key_iv: "0311e568b94b52a5b84853d36ecc1cef"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.042232 32595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "87e35e733b7d487ab250392f019874eb"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5cf9cb630d2b13ce87d7df2b53e9ccd5"
server_key_iv: "0311e568b94b52a5b84853d36ecc1cef"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.046276 32595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:44:26.049026 32610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.050485 32595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:26.050664 32595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "87e35e733b7d487ab250392f019874eb"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5cf9cb630d2b13ce87d7df2b53e9ccd5"
server_key_iv: "0311e568b94b52a5b84853d36ecc1cef"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.050806 32595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:26.069437 32595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:26.072889 32595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:26.073128 32595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:26.082134 32595 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:40063
I20260928 16:44:26.082149 32673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:40063 every 8 connection(s)
I20260928 16:44:26.083199 32595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:26.086243 32674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:26.086585 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32595
I20260928 16:44:26.086722 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:26.087018 31504 external_mini_cluster.cc:1428] Setting key 76d3e149270139e4adfdf50179c3e6ff
I20260928 16:44:26.093873 32674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Bootstrap starting.
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:26.097009 32674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:26.098102 32674 log.cc:824] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:26.100842 32674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: No bootstrap required, opened a new log
I20260928 16:44:26.105034 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.088469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.30.196.62:40063)
0928 16:44:26.088927 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.088958 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.088997 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:44:26.089745 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.089763 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.090119 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.090423 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.091342 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.092511 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.093726 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.094303 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.094443 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.097549 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.097556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.097575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.097647 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.100701 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.101658 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.101664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.101671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.101832 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.102355 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.102359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.102362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.102984 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.103210 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.103497 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.103904 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260928 16:44:26.105587 32674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } }
I20260928 16:44:26.105927 32674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:26.106004 32674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e35e733b7d487ab250392f019874eb, State: Initialized, Role: FOLLOWER
I20260928 16:44:26.106508 32674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } }
I20260928 16:44:26.106653 32674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:26.106752 32674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:26.106896 32674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:26.108204 32674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } }
I20260928 16:44:26.108598 32674 leader_election.cc:302] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87e35e733b7d487ab250392f019874eb; no voters:
I20260928 16:44:26.108981 32674 leader_election.cc:288] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:26.109210 32679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:26.109473 32679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [term 1 LEADER]: Becoming Leader. State: Replica: 87e35e733b7d487ab250392f019874eb, State: Running, Role: LEADER
I20260928 16:44:26.109817 32679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } }
I20260928 16:44:26.110440 32674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:26.111719 32681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87e35e733b7d487ab250392f019874eb. Latest consensus state: current_term: 1 leader_uuid: "87e35e733b7d487ab250392f019874eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } } }
I20260928 16:44:26.111744 32680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87e35e733b7d487ab250392f019874eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e35e733b7d487ab250392f019874eb" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 40063 } } }
I20260928 16:44:26.111901 32680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:26.111901 32681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:26.112375 32688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:26.115871 32688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:26.121596 32688 catalog_manager.cc:1347] Generated new cluster ID: d75015f9b7dc41a9ad5a9f568bc64ad8
I20260928 16:44:26.121713 32688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:26.135100 32688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:26.136989 32688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:26.143877 32688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Generated new TSK 0
I20260928 16:44:26.144759 32688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:26.191357 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:40063
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:26.315706 32700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:26.316177 32700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:26.316301 32700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:26.320880 32700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:26.320950 32700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:26.320993 32700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:26.321060 32700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:26.328692 32700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:40063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:26.329916 32700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:26.330953 32700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:26.341655 32706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.341712 32705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.341655 32708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:26.341717 32700 server_base.cc:1034] running on GCE node
I20260928 16:44:26.342577 32700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:26.343837 32700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:26.345114 32700 hybrid_clock.cc:648] HybridClock initialized: now 1790613866345061 us; error 94 us; skew 500 ppm
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:26.348217 32700 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:26.349247 32700 webserver.cc:460] Webserver started at http://127.30.196.1:40405/ using document root <none> and password file <none>
I20260928 16:44:26.349799 32700 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:26.349860 32700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:26.350054 32700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:26.352206 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "15e95fe50d8e409ba9b6003b70d21a26"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "2bb6fedea5cbe0c6629d71b83da1749b"
server_key_iv: "f3311f97a4c86f705370aeb98cb0c769"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.352726 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "15e95fe50d8e409ba9b6003b70d21a26"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "2bb6fedea5cbe0c6629d71b83da1749b"
server_key_iv: "f3311f97a4c86f705370aeb98cb0c769"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.356913 32700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:26.360136 32715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.361497 32700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:26.361624 32700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "15e95fe50d8e409ba9b6003b70d21a26"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "2bb6fedea5cbe0c6629d71b83da1749b"
server_key_iv: "f3311f97a4c86f705370aeb98cb0c769"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.361791 32700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:26.373447 32700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:26.377096 32700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:26.377329 32700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:26.378911 32700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:26.378966 32700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.379032 32700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:26.379065 32700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:26.391839 32700 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:44109
I20260928 16:44:26.391921 362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:44109 every 8 connection(s)
I20260928 16:44:26.392794 32700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:26.394412 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.380280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:49165 (local address 127.30.196.62:40063)
0928 16:44:26.380435 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.380439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.381776 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 16:44:26.382952 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.382956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.383030 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.383211 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.384871 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.385496 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.386680 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.386981 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.387111 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.390196 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.390205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.390207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.390234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.392079 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.392637 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.392640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.392643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.392735 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.393099 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.393104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.393106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.393375 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.393473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.394109 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.394256 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:44:26.395196 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.381023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.1:49165)
0928 16:44:26.381589 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.381637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.382645 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.383454 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.383463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.384043 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.384693 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.384716 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.385642 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.385645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.386537 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.386548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.387173 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.387177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.387386 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.388151 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.388179 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.389991 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.392216 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.392222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.392238 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.392512 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.392841 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.392844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.392846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.392994 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.393530 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.393536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.393812 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.393980 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.394260 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":158,"threads_started":1}
I20260928 16:44:26.397050 363 heartbeater.cc:347] Connected to a master server at 127.30.196.62:40063
I20260928 16:44:26.397409 363 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:26.397938 363 heartbeater.cc:511] Master 127.30.196.62:40063 requested a full tablet report, sending...
I20260928 16:44:26.399299 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 32700
I20260928 16:44:26.399441 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:26.399838 31504 external_mini_cluster.cc:1428] Setting key 019cd4f48fe1caec48b75b92178b5eb1
I20260928 16:44:26.399885 32628 ts_manager.cc:194] Registered new tserver with Master: 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109)
I20260928 16:44:26.401844 32628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:49165
I20260928 16:44:26.410125 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.400023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.1:44975)
0928 16:44:26.400704 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.400719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.400828 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.401116 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.401119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.401373 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.401519 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.401528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.402374 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.402377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.403118 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.403146 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.404186 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.404191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.404310 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.405052 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.405107 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.405778 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.408242 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.408245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.408247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.408493 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.408899 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.408902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.408904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.409015 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.409616 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.409618 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.409712 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.409810 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.409946 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":165,"threads_started":1}
I20260928 16:44:26.410279 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.400487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:44975 (local address 127.30.196.62:40063)
0928 16:44:26.400633 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.400638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.400801 (+ 163us) server_negotiation.cc:407] Connection header received
0928 16:44:26.400920 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.400923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.400964 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.401092 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.401629 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.402228 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.403528 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.404035 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.404163 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.406027 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.406034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.406040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.406077 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.408031 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.408613 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.408618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.408622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.408724 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.409131 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.409135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.409138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.409456 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.409574 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.409920 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.410074 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:26.432983 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:40063
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:26.555248 372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:26.555550 372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:26.555630 372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:26.559875 372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:26.559957 372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:26.560039 372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:26.560119 372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:26.565760 372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:40063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:26.566969 372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:26.568076 372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:26.578466 377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.578444 380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.578490 378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:26.578507 372 server_base.cc:1034] running on GCE node
I20260928 16:44:26.579188 372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:26.579972 372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:26.581288 372 hybrid_clock.cc:648] HybridClock initialized: now 1790613866581279 us; error 52 us; skew 500 ppm
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:26.584398 372 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:26.585458 372 webserver.cc:460] Webserver started at http://127.30.196.2:43933/ using document root <none> and password file <none>
I20260928 16:44:26.586071 372 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:26.586167 372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:26.586423 372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:26.588898 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "0017a49f81cc4c8db6b1708a6efb1099"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5e7152f5b2101e0eae19694f9124bed1"
server_key_iv: "a3b9cdda549fb97e6ab036039baa09f5"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.589548 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "0017a49f81cc4c8db6b1708a6efb1099"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5e7152f5b2101e0eae19694f9124bed1"
server_key_iv: "a3b9cdda549fb97e6ab036039baa09f5"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.593815 372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:26.597155 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.598605 372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:26.598760 372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "0017a49f81cc4c8db6b1708a6efb1099"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "5e7152f5b2101e0eae19694f9124bed1"
server_key_iv: "a3b9cdda549fb97e6ab036039baa09f5"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.598889 372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:26.613564 372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:26.617652 372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:26.617882 372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:26.619490 372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:26.619571 372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.619654 372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:26.619737 372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:26.631243 372 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:41623
I20260928 16:44:26.631289 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:41623 every 8 connection(s)
I20260928 16:44:26.632249 372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:26.635866 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.620691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:37125 (local address 127.30.196.62:40063)
0928 16:44:26.620862 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.620867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.621852 (+ 985us) server_negotiation.cc:407] Connection header received
0928 16:44:26.623218 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.623224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.623294 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.623411 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.625130 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.625731 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.627322 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.627583 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.627743 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.631618 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.631627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.631629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.631659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.633618 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.634186 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.634189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.634191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.634279 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.634635 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.634639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.634641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.634849 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.634958 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.635493 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.635646 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:44:26.636687 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.621131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.2:37125)
0928 16:44:26.621657 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.621714 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.622915 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.623607 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.623614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.624209 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.624957 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.624976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.625879 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.625883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.627179 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.627190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.627806 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.627812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.628029 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.628720 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.628739 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.631373 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.633755 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.633761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.633776 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.634053 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.634386 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.634389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.634391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.634534 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.634972 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.634981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.635251 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.635452 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.635830 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":159,"threads_started":1}
I20260928 16:44:26.638446 504 heartbeater.cc:347] Connected to a master server at 127.30.196.62:40063
I20260928 16:44:26.638644 504 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:26.639225 504 heartbeater.cc:511] Master 127.30.196.62:40063 requested a full tablet report, sending...
I20260928 16:44:26.640786 32627 ts_manager.cc:194] Registered new tserver with Master: 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2:41623)
I20260928 16:44:26.640944 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 372
I20260928 16:44:26.641039 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:26.641323 31504 external_mini_cluster.cc:1428] Setting key 745b78df983a342484334365bb0e94fb
I20260928 16:44:26.641641 32627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:37125
I20260928 16:44:26.650647 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.640871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.2:54621)
0928 16:44:26.641206 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.641218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.641335 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.641601 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.641604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.641845 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.641968 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.642012 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.643088 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.643093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.644051 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.644066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.644563 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.644570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.644735 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.645695 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.645709 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.646061 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.648627 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.648631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.648633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.648892 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.649307 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.649310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.649311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.649404 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.650136 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.650138 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.650231 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.650331 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.650466 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260928 16:44:26.650768 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.640926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:54621 (local address 127.30.196.62:40063)
0928 16:44:26.641049 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.641053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.641291 (+ 238us) server_negotiation.cc:407] Connection header received
0928 16:44:26.641444 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.641448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.641495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.641579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.642173 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.642946 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.644209 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.644422 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.644507 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.646232 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.646239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.646242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.646273 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.648479 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.649044 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.649050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.649055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.649173 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.649530 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.649536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.649539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.649916 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.650058 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.650428 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.650570 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:26.673221 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:40063
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:26.794723 513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:26.795043 513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:26.795107 513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:26.799512 513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:26.799619 513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:26.799731 513 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:26.799783 513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:26.805544 513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:40063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:26.806938 513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:26.808068 513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:26.818684 518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:26.818717 519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:26.818825 513 server_base.cc:1034] running on GCE node
W20260928 16:44:26.818677 521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:26.819509 513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:26.820298 513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:26.821523 513 hybrid_clock.cc:648] HybridClock initialized: now 1790613866821483 us; error 54 us; skew 500 ppm
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:26.825073 513 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:26.826102 513 webserver.cc:460] Webserver started at http://127.30.196.3:45449/ using document root <none> and password file <none>
I20260928 16:44:26.826750 513 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:26.826849 513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:26.827077 513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:26.829284 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "8ac70398fc693e89940dc10a93c98cd6"
server_key_iv: "4763318126f8e82dc9fbbae0d8a61b6d"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.829823 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "8ac70398fc693e89940dc10a93c98cd6"
server_key_iv: "4763318126f8e82dc9fbbae0d8a61b6d"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.834306 513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:26.837751 529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.839864 513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:44:26.840005 513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
format_stamp: "Formatted at 2026-09-28 16:44:26 on dist-test-slave-5rg1"
server_key: "8ac70398fc693e89940dc10a93c98cd6"
server_key_iv: "4763318126f8e82dc9fbbae0d8a61b6d"
server_key_version: "encryptionkey@0"
I20260928 16:44:26.840152 513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:26.860090 513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:26.864199 513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:26.864399 513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:26.866139 513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:26.866214 513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:26.866294 513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:26.866330 513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:26.878552 513 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:42279
I20260928 16:44:26.878672 646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:42279 every 8 connection(s)
I20260928 16:44:26.879822 513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:26.881421 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 513
I20260928 16:44:26.881592 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:26.881877 31504 external_mini_cluster.cc:1428] Setting key a0ed29b2d64314a3be27eb20b9e3a6fc
I20260928 16:44:26.882735 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.867187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:56091 (local address 127.30.196.62:40063)
0928 16:44:26.867377 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.867383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.868359 (+ 976us) server_negotiation.cc:407] Connection header received
0928 16:44:26.869500 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.869507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.869590 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.869694 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.872223 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.872909 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.874301 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.874683 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.874793 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.877714 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.877718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.877721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.877755 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.879588 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.880787 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.880791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.880793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.880891 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.881420 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.881425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.881432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.881728 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.881842 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.882423 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.882568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:26.883765 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.867635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.3:56091)
0928 16:44:26.868150 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.868202 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.869250 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.869910 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.869923 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.870770 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.872047 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.872077 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.873062 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.873065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.874169 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.874179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.874844 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.874848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.875038 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.875839 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.875857 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.877516 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.879777 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.879789 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.879801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.880570 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.881055 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.881058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.881061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.881287 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.881840 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.881847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.882160 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.882362 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.882724 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260928 16:44:26.885027 647 heartbeater.cc:347] Connected to a master server at 127.30.196.62:40063
I20260928 16:44:26.885210 647 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:26.885790 647 heartbeater.cc:511] Master 127.30.196.62:40063 requested a full tablet report, sending...
I20260928 16:44:26.886862 32628 ts_manager.cc:194] Registered new tserver with Master: 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:26.887498 32628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:56091
I20260928 16:44:26.896435 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.887985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.30.196.3:39435)
0928 16:44:26.888302 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.888316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.888422 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.888799 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.888803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.889094 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.889258 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.889266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.890255 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.890258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.890852 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.890861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.891345 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.891348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.891420 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.892015 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.892028 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.892397 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:26.894482 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.894485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.894487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.894724 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.895092 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:26.895095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:26.895098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:26.895216 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:26.895798 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:26.895804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:26.895905 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.896031 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.896207 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":132,"threads_started":1}
I20260928 16:44:26.896409 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:26.896550 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.887753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:39435 (local address 127.30.196.62:40063)
0928 16:44:26.887932 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.887936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.888406 (+ 470us) server_negotiation.cc:407] Connection header received
0928 16:44:26.888494 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.888498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.888547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.888646 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.889512 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.890113 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.890983 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.891233 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.891317 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.892640 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.892643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.892646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.892671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.894352 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.894839 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.894843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.894845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.894955 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.895316 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.895318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.895319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.895619 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.895750 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.896250 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.896370 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:44:26 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:26.926082 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.915457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.30.196.62:40063)
0928 16:44:26.915852 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.915857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.915884 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:44:26.915957 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.915961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.916027 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.916199 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.917323 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.917989 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.919314 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.919610 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.919796 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.922574 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:26.922578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:26.922580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:26.922613 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:26.924252 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.924721 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.924725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.924727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.924818 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:26.925131 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:26.925135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:26.925136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:26.925352 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:26.925499 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.925770 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.925899 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 16:44:26.930234 32628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:26.933568 32628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:26.950161 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.944682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.30.196.3:42279)
0928 16:44:26.945472 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.945483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.945502 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:26.945581 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.945585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.945769 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.945949 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:26.946194 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.947291 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.948884 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949430 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949559 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.949738 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.949892 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.949990 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":118,"threads_started":1}
I20260928 16:44:26.950150 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.944192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:42279 (local address 127.0.0.1:52314)
0928 16:44:26.944511 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.944525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.944615 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.945936 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.945939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.945959 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:26.946048 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.946055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.947402 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.947405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.948731 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.948743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949580 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.949690 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.949734 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.949855 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.949987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":61,"thread_start_us":115,"threads_started":1}
I20260928 16:44:26.950453 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.944658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:44109 (local address 127.0.0.1:51846)
0928 16:44:26.944897 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.944910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.944986 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.946114 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.946117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.946134 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:26.946235 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.946242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.947226 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.947229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.948673 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.948683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949959 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.950083 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.950107 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.950181 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.950279 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":16,"thread_start_us":98,"threads_started":1}
I20260928 16:44:26.950758 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.943946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.30.196.2:41623)
0928 16:44:26.944726 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.944736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.944755 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:26.944853 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.944859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.945091 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.945259 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:26.945737 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.946971 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.948865 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949649 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949757 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.949906 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.950407 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.950612 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":141,"threads_started":1}
I20260928 16:44:26.950838 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.943788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:41623 (local address 127.0.0.1:39868)
0928 16:44:26.944202 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.944261 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.944405 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.945457 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.945465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.945490 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:26.945615 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.945620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.947102 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.947106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.948731 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.948743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949968 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.950107 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.950124 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:44:26.950183 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 16:44:26.950704 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":656,"thread_start_us":105,"threads_started":1}
I20260928 16:44:26.952116 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.945090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.30.196.1:44109)
0928 16:44:26.945343 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.945350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.945372 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:26.945450 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.945455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.945970 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.946118 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:26.946362 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.947108 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.948814 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.949649 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.949757 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.949890 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:26.951858 (+ 1968us) server_negotiation.cc:299] Negotiation successful
0928 16:44:26.951976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260928 16:44:26.953907 581 tablet_service.cc:1467] Processing CreateTablet for tablet 95110a5dd9144100b80045075a0d96e7 (DEFAULT_TABLE table=test-table [id=7b52ebb581f047789969d6a10d66f420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:26.955128 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95110a5dd9144100b80045075a0d96e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:26.954846 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 95110a5dd9144100b80045075a0d96e7 (DEFAULT_TABLE table=test-table [id=7b52ebb581f047789969d6a10d66f420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:26.955862 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95110a5dd9144100b80045075a0d96e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:26.956661 438 tablet_service.cc:1467] Processing CreateTablet for tablet 95110a5dd9144100b80045075a0d96e7 (DEFAULT_TABLE table=test-table [id=7b52ebb581f047789969d6a10d66f420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:26.957477 438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95110a5dd9144100b80045075a0d96e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:26.963008 670 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Bootstrap starting.
I20260928 16:44:26.964325 672 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Bootstrap starting.
I20260928 16:44:26.966317 670 tablet_bootstrap.cc:654] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:26.966830 672 tablet_bootstrap.cc:654] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:26.967485 670 log.cc:824] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:26.967908 672 log.cc:824] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:26.968211 671 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Bootstrap starting.
I20260928 16:44:26.970036 670 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: No bootstrap required, opened a new log
I20260928 16:44:26.970247 670 ts_tablet_manager.cc:1397] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:44:26.970281 672 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: No bootstrap required, opened a new log
I20260928 16:44:26.970502 672 ts_tablet_manager.cc:1397] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 16:44:26.970494 671 tablet_bootstrap.cc:654] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:26.971365 671 log.cc:824] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:26.973685 671 tablet_bootstrap.cc:492] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: No bootstrap required, opened a new log
I20260928 16:44:26.973893 671 ts_tablet_manager.cc:1397] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:44:26.974159 672 raft_consensus.cc:348] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.974417 672 raft_consensus.cc:374] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:26.974483 672 raft_consensus.cc:729] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0017a49f81cc4c8db6b1708a6efb1099, State: Initialized, Role: FOLLOWER
I20260928 16:44:26.974989 672 consensus_queue.cc:260] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.974937 670 raft_consensus.cc:348] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.975510 670 raft_consensus.cc:374] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:26.975621 670 raft_consensus.cc:729] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e95fe50d8e409ba9b6003b70d21a26, State: Initialized, Role: FOLLOWER
I20260928 16:44:26.976123 672 ts_tablet_manager.cc:1428] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 16:44:26.976442 504 heartbeater.cc:503] Master 127.30.196.62:40063 was elected leader, sending a full tablet report...
I20260928 16:44:26.976346 670 consensus_queue.cc:260] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.977080 363 heartbeater.cc:503] Master 127.30.196.62:40063 was elected leader, sending a full tablet report...
I20260928 16:44:26.977389 670 ts_tablet_manager.cc:1428] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20260928 16:44:26.978839 671 raft_consensus.cc:348] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.979172 671 raft_consensus.cc:374] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:26.979240 671 raft_consensus.cc:729] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab2c21ddcf47fb9867ab23637f2af5, State: Initialized, Role: FOLLOWER
I20260928 16:44:26.979776 671 consensus_queue.cc:260] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:26.980809 647 heartbeater.cc:503] Master 127.30.196.62:40063 was elected leader, sending a full tablet report...
I20260928 16:44:26.981102 671 ts_tablet_manager.cc:1428] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:44:27.053825 677 raft_consensus.cc:482] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:27.054131 677 raft_consensus.cc:504] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.055375 677 leader_election.cc:288] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2:41623), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.061462 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.055745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:44599 (local address 127.30.196.3:42279)
0928 16:44:27.055884 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.055889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.056168 (+ 279us) server_negotiation.cc:407] Connection header received
0928 16:44:27.056320 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.056323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.056373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.056456 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:27.057028 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.057861 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060067 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.060702 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060847 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.060903 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.061242 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.061324 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:27.061482 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.055808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:51067 (local address 127.30.196.2:41623)
0928 16:44:27.055937 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.055943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.056239 (+ 296us) server_negotiation.cc:407] Connection header received
0928 16:44:27.056318 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.056321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.056373 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.056469 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:27.057028 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.057874 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060067 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.060727 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060847 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.060903 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.061242 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.061323 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:27.061506 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.055608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:41623 (local address 127.30.196.1:51067)
0928 16:44:27.056074 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.056086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.056214 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.056578 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.056582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.056613 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:27.056865 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.056873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.058030 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.058033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.059859 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.059871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.060862 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.061046 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.061100 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.061189 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.061297 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":324,"spinlock_wait_cycles":13952,"thread_start_us":190,"threads_started":1}
I20260928 16:44:27.061506 679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.055679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:42279 (local address 127.30.196.1:44599)
0928 16:44:27.056061 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.056075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.056213 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.056578 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.056582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.056613 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:27.056865 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.056873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.058030 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.058033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.059859 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.059871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.060862 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.060866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.061073 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.061108 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.061208 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.061297 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":375,"spinlock_wait_cycles":29824,"thread_start_us":127,"threads_started":1}
I20260928 16:44:27.062417 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95110a5dd9144100b80045075a0d96e7" candidate_uuid: "15e95fe50d8e409ba9b6003b70d21a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0017a49f81cc4c8db6b1708a6efb1099" is_pre_election: true
I20260928 16:44:27.062417 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95110a5dd9144100b80045075a0d96e7" candidate_uuid: "15e95fe50d8e409ba9b6003b70d21a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" is_pre_election: true
I20260928 16:44:27.062779 458 raft_consensus.cc:2454] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e95fe50d8e409ba9b6003b70d21a26 in term 0.
I20260928 16:44:27.062778 601 raft_consensus.cc:2454] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e95fe50d8e409ba9b6003b70d21a26 in term 0.
I20260928 16:44:27.063383 32718 leader_election.cc:302] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 15e95fe50d8e409ba9b6003b70d21a26, 7eab2c21ddcf47fb9867ab23637f2af5; no voters:
I20260928 16:44:27.063772 677 raft_consensus.cc:2781] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:27.063865 677 raft_consensus.cc:482] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:27.063910 677 raft_consensus.cc:3037] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.064988 677 raft_consensus.cc:504] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.065395 677 leader_election.cc:288] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [CANDIDATE]: Term 1 election: Requested vote from peers 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2:41623), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.065754 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95110a5dd9144100b80045075a0d96e7" candidate_uuid: "15e95fe50d8e409ba9b6003b70d21a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
I20260928 16:44:27.065802 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95110a5dd9144100b80045075a0d96e7" candidate_uuid: "15e95fe50d8e409ba9b6003b70d21a26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0017a49f81cc4c8db6b1708a6efb1099"
I20260928 16:44:27.065908 458 raft_consensus.cc:3037] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.065908 601 raft_consensus.cc:3037] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.067248 458 raft_consensus.cc:2454] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e95fe50d8e409ba9b6003b70d21a26 in term 1.
I20260928 16:44:27.067248 601 raft_consensus.cc:2454] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e95fe50d8e409ba9b6003b70d21a26 in term 1.
I20260928 16:44:27.067584 32719 leader_election.cc:302] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 15e95fe50d8e409ba9b6003b70d21a26; no voters:
I20260928 16:44:27.067785 677 raft_consensus.cc:2781] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:27.068190 677 raft_consensus.cc:686] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 LEADER]: Becoming Leader. State: Replica: 15e95fe50d8e409ba9b6003b70d21a26, State: Running, Role: LEADER
I20260928 16:44:27.068559 677 consensus_queue.cc:237] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.072154 32627 catalog_manager.cc:5495] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1). New cstate: current_term: 1 leader_uuid: "15e95fe50d8e409ba9b6003b70d21a26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:27.094475 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.089249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.30.196.1:44109)
0928 16:44:27.089389 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.089394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.089412 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:27.089625 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.089628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.089696 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.089796 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:27.090068 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.090873 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.092436 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.092824 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.092932 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.093403 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.093976 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.094091 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.094255 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.094323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:44:27.101928 601 raft_consensus.cc:1261] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Refusing update from remote peer 15e95fe50d8e409ba9b6003b70d21a26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.101999 458 raft_consensus.cc:1261] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Refusing update from remote peer 15e95fe50d8e409ba9b6003b70d21a26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.102767 681 consensus_queue.cc:1035] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.102978 677 consensus_queue.cc:1035] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.126827 32627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:27.127041 32627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:27.130330 32627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Sending DeleteTablet for 3 replicas of tablet 95110a5dd9144100b80045075a0d96e7
I20260928 16:44:27.131927 32765 tablet_service.cc:1514] Processing DeleteTablet for tablet 95110a5dd9144100b80045075a0d96e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:51846
I20260928 16:44:27.131834 581 tablet_service.cc:1514] Processing DeleteTablet for tablet 95110a5dd9144100b80045075a0d96e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:52314
I20260928 16:44:27.132613 694 tablet_replica.cc:330] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: stopping tablet replica
I20260928 16:44:27.132843 695 tablet_replica.cc:330] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: stopping tablet replica
I20260928 16:44:27.132730 438 tablet_service.cc:1514] Processing DeleteTablet for tablet 95110a5dd9144100b80045075a0d96e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39868
I20260928 16:44:27.133061 694 raft_consensus.cc:2229] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:27.133148 695 raft_consensus.cc:2229] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:27.133520 695 raft_consensus.cc:2258] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.133621 694 raft_consensus.cc:2258] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.133996 696 tablet_replica.cc:330] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: stopping tablet replica
I20260928 16:44:27.134611 696 raft_consensus.cc:2229] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:27.134950 696 raft_consensus.cc:2258] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.134860 32627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:27.135375 32627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:27.135694 695 ts_tablet_manager.cc:1905] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.135959 694 ts_tablet_manager.cc:1905] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.136559 696 ts_tablet_manager.cc:1905] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.138914 695 ts_tablet_manager.cc:1918] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:27.139101 695 log.cc:1196] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/95110a5dd9144100b80045075a0d96e7
I20260928 16:44:27.139029 694 ts_tablet_manager.cc:1918] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:27.139379 694 log.cc:1196] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/95110a5dd9144100b80045075a0d96e7
I20260928 16:44:27.139477 695 ts_tablet_manager.cc:1939] T 95110a5dd9144100b80045075a0d96e7 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting consensus metadata
I20260928 16:44:27.139456 696 ts_tablet_manager.cc:1918] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:27.139555 696 log.cc:1196] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/95110a5dd9144100b80045075a0d96e7
I20260928 16:44:27.139920 694 ts_tablet_manager.cc:1939] T 95110a5dd9144100b80045075a0d96e7 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting consensus metadata
I20260928 16:44:27.139957 696 ts_tablet_manager.cc:1939] T 95110a5dd9144100b80045075a0d96e7 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting consensus metadata
I20260928 16:44:27.140995 32613 catalog_manager.cc:4841] TS 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279): tablet 95110a5dd9144100b80045075a0d96e7 (table test-table [id=7b52ebb581f047789969d6a10d66f420]) successfully deleted
I20260928 16:44:27.141052 32614 catalog_manager.cc:4841] TS 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109): tablet 95110a5dd9144100b80045075a0d96e7 (table test-table [id=7b52ebb581f047789969d6a10d66f420]) successfully deleted
I20260928 16:44:27.141582 32614 catalog_manager.cc:4841] TS 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2:41623): tablet 95110a5dd9144100b80045075a0d96e7 (table test-table [id=7b52ebb581f047789969d6a10d66f420]) successfully deleted
I20260928 16:44:27.141917 581 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (DEFAULT_TABLE table=test-table [id=8abdaa4d0aad42f290159d76d9296056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:27.142212 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.142241 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (DEFAULT_TABLE table=test-table [id=8abdaa4d0aad42f290159d76d9296056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:27.142388 438 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (DEFAULT_TABLE table=test-table [id=8abdaa4d0aad42f290159d76d9296056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:27.142500 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.142630 438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.144753 671 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Bootstrap starting.
I20260928 16:44:27.145188 670 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Bootstrap starting.
I20260928 16:44:27.145820 671 tablet_bootstrap.cc:654] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.146246 670 tablet_bootstrap.cc:654] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.146407 672 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Bootstrap starting.
I20260928 16:44:27.147126 671 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: No bootstrap required, opened a new log
I20260928 16:44:27.147217 671 ts_tablet_manager.cc:1397] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:27.147439 672 tablet_bootstrap.cc:654] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.147773 671 raft_consensus.cc:348] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.147893 671 raft_consensus.cc:374] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.147931 671 raft_consensus.cc:729] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab2c21ddcf47fb9867ab23637f2af5, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.148072 671 consensus_queue.cc:260] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.148370 671 ts_tablet_manager.cc:1428] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:27.148902 670 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: No bootstrap required, opened a new log
I20260928 16:44:27.149019 670 ts_tablet_manager.cc:1397] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 16:44:27.149302 672 tablet_bootstrap.cc:492] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: No bootstrap required, opened a new log
I20260928 16:44:27.149379 672 ts_tablet_manager.cc:1397] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:27.149500 670 raft_consensus.cc:348] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.149609 670 raft_consensus.cc:374] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.149643 670 raft_consensus.cc:729] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e95fe50d8e409ba9b6003b70d21a26, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.149766 670 consensus_queue.cc:260] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.149840 672 raft_consensus.cc:348] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.149955 672 raft_consensus.cc:374] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.150003 672 raft_consensus.cc:729] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0017a49f81cc4c8db6b1708a6efb1099, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.150040 670 ts_tablet_manager.cc:1428] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:44:27.150128 672 consensus_queue.cc:260] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.150550 672 ts_tablet_manager.cc:1428] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:27.170768 676 raft_consensus.cc:482] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:27.170914 676 raft_consensus.cc:504] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.172258 676 leader_election.cc:288] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.177054 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.172652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:35521 (local address 127.30.196.3:42279)
0928 16:44:27.172778 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.172784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.172859 (+ 75us) server_negotiation.cc:407] Connection header received
0928 16:44:27.173089 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.173095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.173150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.173199 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:27.173570 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.174078 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.175508 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.176220 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.176351 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.176409 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.176822 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.176912 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:27.177074 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.172879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:53361 (local address 127.30.196.1:44109)
0928 16:44:27.173026 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.173031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.173048 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:27.173104 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.173107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.173156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.173244 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:27.173569 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.174447 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.175848 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.176355 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.176469 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.176524 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.176851 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.176935 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:27.177580 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.172476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:42279 (local address 127.30.196.2:35521)
0928 16:44:27.172776 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.172792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.172938 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.173358 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.173360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.173372 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:27.173456 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.173461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.174194 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.174197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.175370 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.175380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.176536 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.176540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.176661 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.176688 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.177209 (+ 521us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.177299 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260928 16:44:27.177587 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.172503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:44109 (local address 127.30.196.2:53361)
0928 16:44:27.172927 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.172938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.173012 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.173277 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.173280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.173301 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:27.173428 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.173438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.174579 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.174582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.175717 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.175727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.176608 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.176612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.176703 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.176725 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.177281 (+ 556us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.177365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260928 16:44:27.178049 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6a02f60b1b4a7ca17aa1f7f822d6d5" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" is_pre_election: true
I20260928 16:44:27.178210 601 raft_consensus.cc:2454] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 0.
I20260928 16:44:27.178314 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6a02f60b1b4a7ca17aa1f7f822d6d5" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e95fe50d8e409ba9b6003b70d21a26" is_pre_election: true
I20260928 16:44:27.178582 317 raft_consensus.cc:2454] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 0.
I20260928 16:44:27.178581 390 leader_election.cc:302] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 7eab2c21ddcf47fb9867ab23637f2af5; no voters:
I20260928 16:44:27.178850 676 raft_consensus.cc:2781] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:27.178931 676 raft_consensus.cc:482] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:27.178987 676 raft_consensus.cc:3037] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.180125 676 raft_consensus.cc:504] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.180527 676 leader_election.cc:288] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 election: Requested vote from peers 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.180868 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6a02f60b1b4a7ca17aa1f7f822d6d5" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
I20260928 16:44:27.180868 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6a02f60b1b4a7ca17aa1f7f822d6d5" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e95fe50d8e409ba9b6003b70d21a26"
I20260928 16:44:27.180981 317 raft_consensus.cc:3037] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.180994 601 raft_consensus.cc:3037] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.182192 317 raft_consensus.cc:2454] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 1.
I20260928 16:44:27.182184 601 raft_consensus.cc:2454] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 1.
I20260928 16:44:27.182554 390 leader_election.cc:302] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 7eab2c21ddcf47fb9867ab23637f2af5; no voters:
I20260928 16:44:27.182734 676 raft_consensus.cc:2781] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:27.182917 676 raft_consensus.cc:686] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 LEADER]: Becoming Leader. State: Replica: 0017a49f81cc4c8db6b1708a6efb1099, State: Running, Role: LEADER
I20260928 16:44:27.183187 676 consensus_queue.cc:237] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.185828 32627 catalog_manager.cc:5495] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:44:27 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:27.232012 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.221787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:40063 (local address 127.0.0.1:56504)
0928 16:44:27.222129 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.222146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.222277 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.222573 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.222577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.223139 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:27.223275 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.223284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.224269 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.224273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.225273 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.225284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.225761 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.225765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.225863 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.226585 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:27.226602 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:27.228427 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:27.230328 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:27.230341 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:27.230349 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:27.230591 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:27.230875 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:27.230879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:27.230881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:27.230968 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:27.231360 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:27.231371 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:27.231482 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.231607 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.231798 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260928 16:44:27.232051 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.221876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.30.196.62:40063)
0928 16:44:27.222027 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.222032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.222240 (+ 208us) server_negotiation.cc:407] Connection header received
0928 16:44:27.222379 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.222381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.222424 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.222516 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.223434 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.224142 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.225405 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.225635 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.225723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.228591 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:27.228594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:27.228596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:27.228615 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:27.230190 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.230698 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.230703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.230705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.230782 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.231066 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.231072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.231074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.231253 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:27.231360 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.231654 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.231859 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":69}
I20260928 16:44:27.236845 32627 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:56504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:44:27.237887 32627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:27.244529 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 08f73794fa6b45e694909175cce17829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3600e785e0bf4d638fd8245324fd4ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:27.244529 438 tablet_service.cc:1467] Processing CreateTablet for tablet 08f73794fa6b45e694909175cce17829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3600e785e0bf4d638fd8245324fd4ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:27.244706 581 tablet_service.cc:1467] Processing CreateTablet for tablet 08f73794fa6b45e694909175cce17829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3600e785e0bf4d638fd8245324fd4ceb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:27.244938 438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f73794fa6b45e694909175cce17829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.244938 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f73794fa6b45e694909175cce17829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.245113 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f73794fa6b45e694909175cce17829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:27.247920 672 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: Bootstrap starting.
I20260928 16:44:27.248080 670 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: Bootstrap starting.
I20260928 16:44:27.248955 672 tablet_bootstrap.cc:654] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.248955 670 tablet_bootstrap.cc:654] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.249636 671 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: Bootstrap starting.
I20260928 16:44:27.250576 671 tablet_bootstrap.cc:654] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:27.252031 671 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: No bootstrap required, opened a new log
I20260928 16:44:27.252141 671 ts_tablet_manager.cc:1397] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:27.252319 670 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: No bootstrap required, opened a new log
I20260928 16:44:27.252408 670 ts_tablet_manager.cc:1397] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 16:44:27.252693 672 tablet_bootstrap.cc:492] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: No bootstrap required, opened a new log
I20260928 16:44:27.252771 672 ts_tablet_manager.cc:1397] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:44:27.252701 671 raft_consensus.cc:348] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.252864 671 raft_consensus.cc:374] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.252908 671 raft_consensus.cc:729] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab2c21ddcf47fb9867ab23637f2af5, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.252898 670 raft_consensus.cc:348] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.253003 670 raft_consensus.cc:374] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.253036 670 raft_consensus.cc:729] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e95fe50d8e409ba9b6003b70d21a26, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.253080 671 consensus_queue.cc:260] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.253243 670 consensus_queue.cc:260] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.253345 672 raft_consensus.cc:348] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.253345 678 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.253453 672 raft_consensus.cc:374] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:27.253489 672 raft_consensus.cc:729] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0017a49f81cc4c8db6b1708a6efb1099, State: Initialized, Role: FOLLOWER
I20260928 16:44:27.253499 678 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.253541 689 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.253616 672 consensus_queue.cc:260] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.253680 689 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.253688 670 ts_tablet_manager.cc:1428] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:44:27.253887 672 ts_tablet_manager.cc:1428] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:27.253943 676 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.254062 676 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.257406 671 ts_tablet_manager.cc:1428] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 16:44:27.274509 676 raft_consensus.cc:482] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:27.274681 676 raft_consensus.cc:504] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.275549 676 leader_election.cc:288] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.276496 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f73794fa6b45e694909175cce17829" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" is_pre_election: true
I20260928 16:44:27.276464 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f73794fa6b45e694909175cce17829" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e95fe50d8e409ba9b6003b70d21a26" is_pre_election: true
I20260928 16:44:27.276664 317 raft_consensus.cc:2454] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 0.
I20260928 16:44:27.276662 601 raft_consensus.cc:2454] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 0.
I20260928 16:44:27.277051 391 leader_election.cc:302] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 15e95fe50d8e409ba9b6003b70d21a26; no voters:
I20260928 16:44:27.277303 676 raft_consensus.cc:2781] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:27.277386 676 raft_consensus.cc:482] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:27.277427 676 raft_consensus.cc:3037] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.278801 676 raft_consensus.cc:504] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.279273 676 leader_election.cc:288] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 election: Requested vote from peers 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109), 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279)
I20260928 16:44:27.279774 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f73794fa6b45e694909175cce17829" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e95fe50d8e409ba9b6003b70d21a26"
I20260928 16:44:27.279788 601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f73794fa6b45e694909175cce17829" candidate_uuid: "0017a49f81cc4c8db6b1708a6efb1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab2c21ddcf47fb9867ab23637f2af5"
I20260928 16:44:27.279927 601 raft_consensus.cc:3037] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.279932 317 raft_consensus.cc:3037] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:27.281157 601 raft_consensus.cc:2454] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 1.
I20260928 16:44:27.281361 317 raft_consensus.cc:2454] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0017a49f81cc4c8db6b1708a6efb1099 in term 1.
I20260928 16:44:27.281502 390 leader_election.cc:302] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0017a49f81cc4c8db6b1708a6efb1099, 7eab2c21ddcf47fb9867ab23637f2af5; no voters:
I20260928 16:44:27.281693 676 raft_consensus.cc:2781] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:27.282047 676 raft_consensus.cc:686] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 LEADER]: Becoming Leader. State: Replica: 0017a49f81cc4c8db6b1708a6efb1099, State: Running, Role: LEADER
I20260928 16:44:27.282253 676 consensus_queue.cc:237] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } }
I20260928 16:44:27.283490 702 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: TxnStatusTablet state changed. Reason: New leader 0017a49f81cc4c8db6b1708a6efb1099. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.283608 702 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:27.283969 714 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:44:27.284138 714 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:44:27.284668 32627 catalog_manager.cc:5495] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:27.300004 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.295855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.30.196.2:41623)
0928 16:44:27.295990 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.295994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.296007 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:27.296089 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.296091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.296146 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.296253 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:27.296533 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.297191 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.298307 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.298537 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.298656 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.298984 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.299530 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.299657 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.299774 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.299849 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:27.300004 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.295474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:41623 (local address 127.0.0.1:45242)
0928 16:44:27.295692 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.295711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.295868 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.296282 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.296285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.296293 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:27.296421 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.296427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.297309 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.297313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.298174 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.298185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.298662 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.298665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.298763 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.298826 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.299632 (+ 806us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.299651 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.299763 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.299862 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 16:44:27.303402 601 raft_consensus.cc:1261] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Refusing update from remote peer 0017a49f81cc4c8db6b1708a6efb1099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.303611 317 raft_consensus.cc:1261] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Refusing update from remote peer 0017a49f81cc4c8db6b1708a6efb1099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.304127 676 consensus_queue.cc:1035] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.304359 702 consensus_queue.cc:1035] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.306430 689 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: TxnStatusTablet state changed. Reason: New leader 0017a49f81cc4c8db6b1708a6efb1099. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.306557 689 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.306701 678 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: TxnStatusTablet state changed. Reason: New leader 0017a49f81cc4c8db6b1708a6efb1099. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.306821 678 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.308578 702 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.308710 702 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:27.308851 686 mvcc.cc:205] Tried to move back new op lower bound from 7334354400469078016 to 7334354400389840896. Current Snapshot: MvccSnapshot[applied={T|T < 7334354400469078016 or (T in {7334354400469078016})}]
I20260928 16:44:27.311365 685 mvcc.cc:205] Tried to move back new op lower bound from 7334354400469078016 to 7334354400389840896. Current Snapshot: MvccSnapshot[applied={T|T < 7334354400469078016 or (T in {7334354400469078016})}]
I20260928 16:44:27.312044 678 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.312186 678 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 7eab2c21ddcf47fb9867ab23637f2af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.315320 676 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.315470 676 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 0017a49f81cc4c8db6b1708a6efb1099: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:27.317010 684 mvcc.cc:205] Tried to move back new op lower bound from 7334354400469078016 to 7334354400389840896. Current Snapshot: MvccSnapshot[applied={T|T < 7334354400469078016}]
I20260928 16:44:27.317224 689 tablet_replica.cc:439] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0017a49f81cc4c8db6b1708a6efb1099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 } } peers { permanent_uuid: "0017a49f81cc4c8db6b1708a6efb1099" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 41623 } } peers { permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 } } }
I20260928 16:44:27.317349 689 tablet_replica.cc:442] T 08f73794fa6b45e694909175cce17829 P 15e95fe50d8e409ba9b6003b70d21a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:27.323593 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.318776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.30.196.2:41623)
0928 16:44:27.319059 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.319064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.319082 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:27.319144 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.319147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.319194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.319272 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:27.319698 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.320452 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.321523 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.321725 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.322150 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.322687 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.322871 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.322999 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.323157 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.323247 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260928 16:44:27.335019 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.331572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:59093 (local address 127.30.196.2:41623)
0928 16:44:27.331726 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.331731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.331965 (+ 234us) server_negotiation.cc:407] Connection header received
0928 16:44:27.332128 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.332132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.332183 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.332262 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:27.332532 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.333035 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.333919 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.334163 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.334275 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.334557 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.334635 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.334713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.334797 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.334847 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":40}
I20260928 16:44:27.335033 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.331551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:41623 (local address 127.30.196.2:59093)
0928 16:44:27.331873 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.331894 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.332024 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.332270 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.332273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.332279 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:27.332419 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.332428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.333141 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.333146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.333793 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.333800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.334277 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.334280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.334378 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.334425 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:27.334717 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:27.334725 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:44:27.334789 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 16:44:27.334873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":9,"thread_start_us":139,"threads_started":1}
I20260928 16:44:27.342459 601 raft_consensus.cc:1261] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Refusing update from remote peer 0017a49f81cc4c8db6b1708a6efb1099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.342460 317 raft_consensus.cc:1261] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Refusing update from remote peer 0017a49f81cc4c8db6b1708a6efb1099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:27.342929 676 consensus_queue.cc:1035] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e95fe50d8e409ba9b6003b70d21a26" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.343135 702 consensus_queue.cc:1035] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eab2c21ddcf47fb9867ab23637f2af5" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:27.393023 32628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:27.393208 32628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:27.396212 32628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87e35e733b7d487ab250392f019874eb: Sending DeleteTablet for 3 replicas of tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5
I20260928 16:44:27.397022 32765 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:51846
I20260928 16:44:27.397187 581 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:52314
I20260928 16:44:27.397243 694 tablet_replica.cc:330] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: stopping tablet replica
I20260928 16:44:27.397372 695 tablet_replica.cc:330] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: stopping tablet replica
I20260928 16:44:27.397372 694 raft_consensus.cc:2229] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:27.397481 695 raft_consensus.cc:2229] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:27.397504 694 raft_consensus.cc:2258] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.397593 695 raft_consensus.cc:2258] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.397711 437 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:27 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39868
I20260928 16:44:27.397892 696 tablet_replica.cc:330] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: stopping tablet replica
I20260928 16:44:27.398013 696 raft_consensus.cc:2229] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:27.398383 696 raft_consensus.cc:2258] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:27.398432 695 ts_tablet_manager.cc:1905] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.399260 694 ts_tablet_manager.cc:1905] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.399260 696 ts_tablet_manager.cc:1905] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:27.401690 695 ts_tablet_manager.cc:1918] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:27.401770 695 log.cc:1196] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/6d6a02f60b1b4a7ca17aa1f7f822d6d5
I20260928 16:44:27.402034 695 ts_tablet_manager.cc:1939] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 7eab2c21ddcf47fb9867ab23637f2af5: Deleting consensus metadata
I20260928 16:44:27.402181 694 ts_tablet_manager.cc:1918] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:27.402263 694 log.cc:1196] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/6d6a02f60b1b4a7ca17aa1f7f822d6d5
I20260928 16:44:27.402518 694 ts_tablet_manager.cc:1939] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 15e95fe50d8e409ba9b6003b70d21a26: Deleting consensus metadata
I20260928 16:44:27.402537 696 ts_tablet_manager.cc:1918] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:27.402762 696 log.cc:1196] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/6d6a02f60b1b4a7ca17aa1f7f822d6d5
I20260928 16:44:27.402832 32613 catalog_manager.cc:4841] TS 7eab2c21ddcf47fb9867ab23637f2af5 (127.30.196.3:42279): tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (table test-table [id=8abdaa4d0aad42f290159d76d9296056]) successfully deleted
I20260928 16:44:27.403152 696 ts_tablet_manager.cc:1939] T 6d6a02f60b1b4a7ca17aa1f7f822d6d5 P 0017a49f81cc4c8db6b1708a6efb1099: Deleting consensus metadata
I20260928 16:44:27.404306 32614 catalog_manager.cc:4841] TS 0017a49f81cc4c8db6b1708a6efb1099 (127.30.196.2:41623): tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (table test-table [id=8abdaa4d0aad42f290159d76d9296056]) successfully deleted
I20260928 16:44:27.404831 32614 catalog_manager.cc:4841] TS 15e95fe50d8e409ba9b6003b70d21a26 (127.30.196.1:44109): tablet 6d6a02f60b1b4a7ca17aa1f7f822d6d5 (table test-table [id=8abdaa4d0aad42f290159d76d9296056]) successfully deleted
Sep 28 16:44:27 dist-test-slave-5rg1 krb5kdc[32582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:27.416145 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.403212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.30.196.1:44109)
0928 16:44:27.403325 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.403330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.404970 (+ 1640us) server_negotiation.cc:407] Connection header received
0928 16:44:27.405218 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.405257 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.405312 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.405425 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.406135 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.406662 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.407513 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.407830 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.407941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.411751 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:27.411764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:27.411780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:27.411808 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:27.413986 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.414481 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.414487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.414489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.414582 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.414873 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.414877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.414879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.415228 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:27.415458 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.415729 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.415878 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 16:44:27.416898 357 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55826
I20260928 16:44:27.426874 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.418633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.30.196.62:40063)
0928 16:44:27.418763 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.418767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.419191 (+ 424us) server_negotiation.cc:407] Connection header received
0928 16:44:27.419319 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.419322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.419368 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.419468 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.420200 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.420741 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.421582 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.421821 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.421946 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.423321 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:27.423325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:27.423328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:27.423355 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:27.424936 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.425374 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.425378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.425380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.425466 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:27.425751 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:27.425755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:27.425757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:27.426114 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:27.426279 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:27.426568 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 16:44:27.426698 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 16:44:27.427472 32628 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53958
I20260928 16:44:27.429780 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32700
I20260928 16:44:27.438495 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 372
I20260928 16:44:27.446168 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 513
I20260928 16:44:27.452742 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 32595
2026-09-28T16:44:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2772 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:27 dist-test-slave-5rg1 krb5kdc[733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:27 dist-test-slave-5rg1 krb5kdc[733](info): set up 2 sockets
Sep 28 16:44:27 dist-test-slave-5rg1 krb5kdc[733](info): commencing operation
krb5kdc: starting...
W20260928 16:44:28.527211 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:28 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:28.680555 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46123
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:28.803342 746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:28.803684 746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:28.803728 746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:28.808579 746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:28.808674 746 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:44:28.808694 746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:28.808713 746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:28.808730 746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:28.808748 746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:28.814919 746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:28.816128 746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:28.817260 746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:28.826406 754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:28.826419 751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:28.826423 752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:28.826784 746 server_base.cc:1034] running on GCE node
I20260928 16:44:28.827591 746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:28.828851 746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:28.830104 746 hybrid_clock.cc:648] HybridClock initialized: now 1790613868830081 us; error 38 us; skew 500 ppm
Sep 28 16:44:28 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613868, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:28.839478 746 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:28.840688 746 webserver.cc:460] Webserver started at http://127.30.196.62:38119/ using document root <none> and password file <none>
I20260928 16:44:28.841316 746 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:28.841416 746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:28.841655 746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:28.843919 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "c0ed19598d834f02a94f7ecc173274e7"
format_stamp: "Formatted at 2026-09-28 16:44:28 on dist-test-slave-5rg1"
server_key: "18622851ab46155d05d388dd15cbb133"
server_key_iv: "6f595bcc738190025e7e502814fb6e83"
server_key_version: "encryptionkey@0"
I20260928 16:44:28.844522 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "c0ed19598d834f02a94f7ecc173274e7"
format_stamp: "Formatted at 2026-09-28 16:44:28 on dist-test-slave-5rg1"
server_key: "18622851ab46155d05d388dd15cbb133"
server_key_iv: "6f595bcc738190025e7e502814fb6e83"
server_key_version: "encryptionkey@0"
I20260928 16:44:28.849999 746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:28.854060 761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:28.856197 746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:44:28.856381 746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "c0ed19598d834f02a94f7ecc173274e7"
format_stamp: "Formatted at 2026-09-28 16:44:28 on dist-test-slave-5rg1"
server_key: "18622851ab46155d05d388dd15cbb133"
server_key_iv: "6f595bcc738190025e7e502814fb6e83"
server_key_version: "encryptionkey@0"
I20260928 16:44:28.856559 746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:28.893931 746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:28.897818 746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:28.898082 746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:28.906505 746 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:46123
I20260928 16:44:28.906548 823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:46123 every 8 connection(s)
I20260928 16:44:28.907543 746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:28.908741 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 746
I20260928 16:44:28.908885 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:28.909217 31504 external_mini_cluster.cc:1428] Setting key 3248027b816c3f772ff9a2f73fe19b19
I20260928 16:44:28.911829 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:28.919781 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Bootstrap starting.
Sep 28 16:44:28 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:28.923789 824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:28.925069 824 log.cc:824] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:28.926843 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:28.910720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.30.196.62:46123)
0928 16:44:28.911270 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:28.911286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:28.911324 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:44:28.912593 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:28.912614 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:28.912964 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:28.913411 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:28.914273 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.915316 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.916352 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.916845 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.917052 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.921042 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:28.921047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:28.921062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:28.921098 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:28.923526 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:28.924190 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:28.924201 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:28.924206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:28.924329 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:28.924658 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:28.924661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:28.924662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:28.925108 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:28.925341 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:28.925666 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 16:44:28.925983 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":164,"threads_started":1}
I20260928 16:44:28.927825 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: No bootstrap required, opened a new log
I20260928 16:44:28.932859 824 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } }
I20260928 16:44:28.933225 824 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:28.933326 824 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ed19598d834f02a94f7ecc173274e7, State: Initialized, Role: FOLLOWER
I20260928 16:44:28.933822 824 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } }
I20260928 16:44:28.933993 824 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:28.934083 824 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:28.934217 824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:28.935302 824 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } }
I20260928 16:44:28.935760 824 leader_election.cc:302] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ed19598d834f02a94f7ecc173274e7; no voters:
I20260928 16:44:28.936280 829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:28.936342 824 leader_election.cc:288] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:28.936558 829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [term 1 LEADER]: Becoming Leader. State: Replica: c0ed19598d834f02a94f7ecc173274e7, State: Running, Role: LEADER
I20260928 16:44:28.936888 829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } }
I20260928 16:44:28.937839 824 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:28.938978 830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ed19598d834f02a94f7ecc173274e7. Latest consensus state: current_term: 1 leader_uuid: "c0ed19598d834f02a94f7ecc173274e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } } }
I20260928 16:44:28.939105 830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:28.939392 831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ed19598d834f02a94f7ecc173274e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed19598d834f02a94f7ecc173274e7" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46123 } } }
I20260928 16:44:28.939481 831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7 [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:28.943590 844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:28.943715 844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:28.943794 838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:28.944476 838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:28.950997 838 catalog_manager.cc:1347] Generated new cluster ID: 584a5920a58140c5829080a2690511f5
I20260928 16:44:28.951098 838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:28.962844 838 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:28.964074 838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:28.974644 838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Generated new TSK 0
I20260928 16:44:28.975574 838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:29.012691 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46123
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:29.140971 850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:29.141446 850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:29.141588 850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:29.146504 850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:29.146586 850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:29.146636 850 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:44:29.146652 850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:29.146688 850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:29.152879 850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:29.154006 850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:29.155071 850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:29.165982 858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:29.166038 855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:29.166010 856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:29.166195 850 server_base.cc:1034] running on GCE node
I20260928 16:44:29.166692 850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:29.167377 850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:29.168574 850 hybrid_clock.cc:648] HybridClock initialized: now 1790613869168551 us; error 37 us; skew 500 ppm
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:29.171785 850 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:29.173156 850 webserver.cc:460] Webserver started at http://127.30.196.1:38103/ using document root <none> and password file <none>
I20260928 16:44:29.173802 850 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:29.173910 850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:29.174178 850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:29.176898 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "34e5561a78a3c513c0f8d127d7cdd59f"
server_key_iv: "a2d9214965b2fb9e944b91d4d7009dac"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.177569 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "34e5561a78a3c513c0f8d127d7cdd59f"
server_key_iv: "a2d9214965b2fb9e944b91d4d7009dac"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.182041 850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:29.185169 865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.186614 850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:29.186792 850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "34e5561a78a3c513c0f8d127d7cdd59f"
server_key_iv: "a2d9214965b2fb9e944b91d4d7009dac"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.186933 850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:29.202733 850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:29.206985 850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:29.207217 850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:29.208994 850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:29.209074 850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.209149 850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:29.209195 850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.220892 850 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:38241
I20260928 16:44:29.221009 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:38241 every 8 connection(s)
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:29.221900 850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:29.225950 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.210807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:35497 (local address 127.30.196.62:46123)
0928 16:44:29.210956 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.210959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.212476 (+ 1517us) server_negotiation.cc:407] Connection header received
0928 16:44:29.214501 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.214508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.214597 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.214711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.216383 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.217087 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.218445 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.218735 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.218855 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.221844 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.221847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.221849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.221875 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.223842 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.224371 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.224373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.224374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.224465 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.224823 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.224826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.224827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.225091 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.225179 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.225693 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.225812 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:44:29.226864 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.211290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.1:35497)
0928 16:44:29.212168 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.212274 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.214132 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.214997 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.215009 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.215510 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.216202 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.216221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.217229 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.217234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.218261 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.218272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.218907 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.218911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.219093 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.219775 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.219795 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.221652 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.223957 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.223962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.223974 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.224256 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.224554 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.224556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.224558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.224700 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.225191 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.225196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.225433 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.225585 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.225854 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":167,"threads_started":1}
I20260928 16:44:29.228340 981 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46123
I20260928 16:44:29.228520 981 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:29.229046 981 heartbeater.cc:511] Master 127.30.196.62:46123 requested a full tablet report, sending...
I20260928 16:44:29.230616 776 ts_manager.cc:194] Registered new tserver with Master: 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241)
I20260928 16:44:29.231175 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 850
I20260928 16:44:29.231299 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:29.231611 31504 external_mini_cluster.cc:1428] Setting key 1ecf7c305289ef39ead2fb0dfde7ffb5
I20260928 16:44:29.232193 776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:35497
I20260928 16:44:29.241510 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.230883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.1:33793)
0928 16:44:29.231361 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.231377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.231482 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.231913 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.231916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.232142 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.232299 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.232308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.233299 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.233302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.234132 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.234142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.235232 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.235238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.235363 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.236173 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.236193 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.236712 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.239351 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.239357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.239362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.239773 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.240128 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.240132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.240134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.240240 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.240714 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.240718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.240840 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.240969 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.241153 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":241,"threads_started":1}
I20260928 16:44:29.241600 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.230950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:33793 (local address 127.30.196.62:46123)
0928 16:44:29.231073 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.231079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.231469 (+ 390us) server_negotiation.cc:407] Connection header received
0928 16:44:29.231603 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.231606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.231705 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.231790 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.232442 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.233146 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.234271 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.234992 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.235178 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.237088 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.237093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.237096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.237140 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.239190 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.239878 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.239881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.239883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.240019 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.240377 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.240380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.240381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.240601 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.240697 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.241174 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.241455 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:29.269326 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46123
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:29.402601 990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:29.402977 990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:29.403033 990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:29.407423 990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:29.407500 990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:29.407545 990 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:44:29.407559 990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:29.407593 990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:29.413354 990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:29.414486 990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:29.415531 990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:29.425814 995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:29.425818 998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:29.425915 996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:29.426424 990 server_base.cc:1034] running on GCE node
I20260928 16:44:29.426931 990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:29.427641 990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:29.429294 990 hybrid_clock.cc:648] HybridClock initialized: now 1790613869429272 us; error 31 us; skew 500 ppm
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:29.432480 990 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:29.433545 990 webserver.cc:460] Webserver started at http://127.30.196.2:46119/ using document root <none> and password file <none>
I20260928 16:44:29.434216 990 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:29.434336 990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:29.434602 990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:29.437006 990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "f05885a8db8d4c10948540bd827faa98"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "92e2a0cd4c47c54ad4e2331926600619"
server_key_iv: "0b5972f540448a2ae42c0702ec31c9e7"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.437635 990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "f05885a8db8d4c10948540bd827faa98"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "92e2a0cd4c47c54ad4e2331926600619"
server_key_iv: "0b5972f540448a2ae42c0702ec31c9e7"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.442320 990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:44:29.445183 1005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.446543 990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:29.446683 990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "f05885a8db8d4c10948540bd827faa98"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "92e2a0cd4c47c54ad4e2331926600619"
server_key_iv: "0b5972f540448a2ae42c0702ec31c9e7"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.446820 990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:29.459323 990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:29.463243 990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:29.463404 990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:29.464901 990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:29.464983 990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.465054 990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:29.465085 990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.476557 990 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:44371
I20260928 16:44:29.476781 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:44371 every 8 connection(s)
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:29.477768 990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:29.482609 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.466660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:46103 (local address 127.30.196.62:46123)
0928 16:44:29.466824 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.466828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.468119 (+ 1291us) server_negotiation.cc:407] Connection header received
0928 16:44:29.469655 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.469660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.469736 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.469824 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.471734 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.472660 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.474491 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.474887 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.475032 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.478543 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.478547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.478550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.478579 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.480310 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.480890 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.480893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.480895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.480982 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.481382 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.481387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.481389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.481629 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.481737 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.482294 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.482430 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:29.483471 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.467210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.2:46103)
0928 16:44:29.467853 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.467993 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.469310 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.470076 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.470089 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.470698 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.471491 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.471517 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.472824 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.472830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.474359 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.474374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.475069 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.475076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.475336 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.476092 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.476113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.478312 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.480449 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.480455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.480470 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.480776 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.481080 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.481083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.481085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.481265 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.481735 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.481759 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.482025 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.482237 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.482591 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":142,"threads_started":1}
I20260928 16:44:29.484753 1121 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46123
I20260928 16:44:29.484910 1121 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:29.485555 1121 heartbeater.cc:511] Master 127.30.196.62:46123 requested a full tablet report, sending...
I20260928 16:44:29.486786 777 ts_manager.cc:194] Registered new tserver with Master: f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:29.487605 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 990
I20260928 16:44:29.487630 777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:46103
I20260928 16:44:29.487743 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:29.488089 31504 external_mini_cluster.cc:1428] Setting key b8c88ae7666def60fec819330c4a2c33
I20260928 16:44:29.495370 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.487355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:56851 (local address 127.30.196.62:46123)
0928 16:44:29.487529 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.487533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.487840 (+ 307us) server_negotiation.cc:407] Connection header received
0928 16:44:29.488031 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.488035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.488084 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.488174 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.488782 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.489453 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.490283 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.490578 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.490693 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.492058 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.492063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.492066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.492107 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.493557 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.494096 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.494098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.494100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.494174 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.494461 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.494466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.494468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.494696 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.494813 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.495081 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.495195 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:29.495389 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.487231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.2:56851)
0928 16:44:29.487756 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.487772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.487917 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.488298 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.488301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.488549 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.488660 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.488666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.489585 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.489588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.490150 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.490158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.490712 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.490716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.490818 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.491433 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.491448 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.491896 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.493680 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.493685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.493687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.493972 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.494270 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.494274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.494276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.494366 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.494804 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.494807 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.494912 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.495025 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.495190 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:29.521193 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46123
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:29.649225 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:29.649544 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:29.649600 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:29.654649 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:29.654743 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:29.654798 1130 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:44:29.654815 1130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:29.654853 1130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:29.661592 1130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:29.662722 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:29.663797 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:29.674998 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:29.674965 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:29.675194 1130 server_base.cc:1034] running on GCE node
W20260928 16:44:29.674978 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:29.675832 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:29.676532 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:29.677735 1130 hybrid_clock.cc:648] HybridClock initialized: now 1790613869677700 us; error 41 us; skew 500 ppm
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:29.680874 1130 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:29.681911 1130 webserver.cc:460] Webserver started at http://127.30.196.3:41655/ using document root <none> and password file <none>
I20260928 16:44:29.682507 1130 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:29.682582 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:29.682770 1130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:29.685150 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "0196bf5438a84233bca0f7d7cbcc9914"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "d1838fd1ca173a6991b94534612ac52a"
server_key_iv: "2effb309d2f5ea09be06b66d841b85b1"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.685848 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "0196bf5438a84233bca0f7d7cbcc9914"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "d1838fd1ca173a6991b94534612ac52a"
server_key_iv: "2effb309d2f5ea09be06b66d841b85b1"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.690002 1130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:29.692803 1145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.694203 1130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:29.694319 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "0196bf5438a84233bca0f7d7cbcc9914"
format_stamp: "Formatted at 2026-09-28 16:44:29 on dist-test-slave-5rg1"
server_key: "d1838fd1ca173a6991b94534612ac52a"
server_key_iv: "2effb309d2f5ea09be06b66d841b85b1"
server_key_version: "encryptionkey@0"
I20260928 16:44:29.694566 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:29.707347 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:29.711135 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:29.711366 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:29.713029 1130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:29.713083 1130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:29.713125 1130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:29.713140 1130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:29.725076 1130 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:33167
I20260928 16:44:29.725143 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:33167 every 8 connection(s)
I20260928 16:44:29.726462 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:29.729975 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.714196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55713 (local address 127.30.196.62:46123)
0928 16:44:29.714370 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.714375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.715302 (+ 927us) server_negotiation.cc:407] Connection header received
0928 16:44:29.716327 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.716330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.716393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.716711 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.718588 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.719446 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.721430 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.721809 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.721906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.725194 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.725198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.725201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.725231 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.727521 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.728210 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.728213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.728215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.728332 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.728699 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.728702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.728704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.728970 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.729094 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.729673 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.729812 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:44:29.730587 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1130
I20260928 16:44:29.730739 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:29.731002 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.714614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.3:55713)
0928 16:44:29.715098 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.715160 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.716098 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.716965 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.716978 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.717613 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.718376 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.718404 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.719589 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.719593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.721267 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.721283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.721969 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.721975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.722211 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.723009 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.723035 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.724977 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.727659 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.727703 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.727719 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.728092 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.728430 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.728435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.728439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.728592 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.729069 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.729076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.729422 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.729613 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.730077 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260928 16:44:29.731098 31504 external_mini_cluster.cc:1428] Setting key fba9a5fbe03d1043bb936f1e4b00ef00
I20260928 16:44:29.732532 1261 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46123
I20260928 16:44:29.732748 1261 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:29.733301 1261 heartbeater.cc:511] Master 127.30.196.62:46123 requested a full tablet report, sending...
I20260928 16:44:29.734730 777 ts_manager.cc:194] Registered new tserver with Master: 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167)
I20260928 16:44:29.735504 777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:55713
I20260928 16:44:29.745062 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.735850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55473 (local address 127.30.196.62:46123)
0928 16:44:29.735974 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.735981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.736318 (+ 337us) server_negotiation.cc:407] Connection header received
0928 16:44:29.736416 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.736419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.736467 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.736552 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.736973 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.737713 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.738611 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.738825 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.738992 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.740731 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.740734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.740737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.740771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.742535 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.743171 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.743174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.743175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.743264 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.743641 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.743644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.743645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.743960 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.744063 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.744751 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.744901 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 16:44:29.745115 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.735805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.30.196.3:55473)
0928 16:44:29.736193 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.736208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.736317 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.736559 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.736561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.736764 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.736863 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.736868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.737867 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.737870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.738484 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.738493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.739032 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.739036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.739118 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.739781 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.739809 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.740557 (+ 748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.742726 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.742732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.742735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.743020 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.743405 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.743408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.743410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.743522 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.744139 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.744142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.744391 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.744601 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.744826 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":215,"threads_started":1}
I20260928 16:44:29.746932 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:29.777396 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.767158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.30.196.62:46123)
0928 16:44:29.767297 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.767301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.767314 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:29.767360 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.767363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.767407 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.767510 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.768397 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.768997 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.769960 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.770203 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.770322 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.773783 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.773789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.773794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.773833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.775378 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.775966 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.775970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.775972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.776067 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.776385 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.776389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.776391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.776608 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.776768 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.777059 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.777210 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:44:29.781143 777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:29.784544 777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:29.802000 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.795656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.0.0.1:55510)
0928 16:44:29.796436 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.796493 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.796626 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.797742 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.797747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.797771 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:29.797869 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.797875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.799048 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.799051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.800363 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.800372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801424 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.801534 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801578 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.801698 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.801826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":92,"threads_started":1}
I20260928 16:44:29.802060 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.795656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33167 (local address 127.0.0.1:53370)
0928 16:44:29.796894 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.796908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.797004 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.797499 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.797505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.797527 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:29.797684 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.797718 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.799212 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.799216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.800619 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.800631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801681 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.801775 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801796 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.801847 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.801902 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":56,"thread_start_us":89,"threads_started":1}
I20260928 16:44:29.802155 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.795880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.30.196.3:33167)
0928 16:44:29.796864 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.796876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.796984 (+ 108us) server_negotiation.cc:407] Connection header received
0928 16:44:29.797141 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.797148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.797344 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.797510 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:29.797842 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.799069 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.800756 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801524 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801674 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801829 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.801894 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.801992 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":119,"threads_started":1}
I20260928 16:44:29.802322 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.796679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.30.196.2:44371)
0928 16:44:29.797264 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.797276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.797298 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:29.797372 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.797377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.797562 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.797720 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:29.798134 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.799212 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.800771 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801295 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801593 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.802068 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.802172 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":103,"threads_started":1}
I20260928 16:44:29.802814 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.795872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.30.196.1:38241)
0928 16:44:29.797267 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.797274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.797298 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:44:29.797372 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.797377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.797562 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.797736 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:29.797997 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.798928 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.800490 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801282 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801402 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801564 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.802558 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.802670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":115,"threads_started":1}
I20260928 16:44:29.803004 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.796488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44371 (local address 127.0.0.1:49122)
0928 16:44:29.797006 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.797020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.797114 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.797869 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.797872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.797887 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:29.798007 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.798014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.799327 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.799331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.800619 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.800631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.801645 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.801649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.801732 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.801754 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.801830 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.801916 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":98,"thread_start_us":94,"threads_started":1}
I20260928 16:44:29.805089 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 7394729bb8a049cb8481bcade1969b92 (DEFAULT_TABLE table=test-table [id=79f3f3f731154842bebc91210a29779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:29.806138 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394729bb8a049cb8481bcade1969b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.805497 915 tablet_service.cc:1467] Processing CreateTablet for tablet 7394729bb8a049cb8481bcade1969b92 (DEFAULT_TABLE table=test-table [id=79f3f3f731154842bebc91210a29779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:29.805509 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 7394729bb8a049cb8481bcade1969b92 (DEFAULT_TABLE table=test-table [id=79f3f3f731154842bebc91210a29779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:29.806787 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394729bb8a049cb8481bcade1969b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.806787 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7394729bb8a049cb8481bcade1969b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.814013 1284 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Bootstrap starting.
I20260928 16:44:29.816425 1285 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Bootstrap starting.
I20260928 16:44:29.816519 1284 tablet_bootstrap.cc:654] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.817619 1286 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Bootstrap starting.
I20260928 16:44:29.818783 1284 log.cc:824] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:29.819953 1286 tablet_bootstrap.cc:654] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.819949 1285 tablet_bootstrap.cc:654] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.820777 1286 log.cc:824] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:29.820785 1285 log.cc:824] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:29.821528 1284 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: No bootstrap required, opened a new log
I20260928 16:44:29.821755 1284 ts_tablet_manager.cc:1397] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:44:29.822751 1285 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: No bootstrap required, opened a new log
I20260928 16:44:29.822949 1285 ts_tablet_manager.cc:1397] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:44:29.825512 1286 tablet_bootstrap.cc:492] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: No bootstrap required, opened a new log
I20260928 16:44:29.826004 1286 ts_tablet_manager.cc:1397] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 16:44:29.826192 1284 raft_consensus.cc:348] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.826500 1284 raft_consensus.cc:374] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.826589 1284 raft_consensus.cc:729] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.827121 1284 consensus_queue.cc:260] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.827656 1285 raft_consensus.cc:348] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.828014 1285 raft_consensus.cc:374] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.828111 1285 raft_consensus.cc:729] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.828092 1261 heartbeater.cc:503] Master 127.30.196.62:46123 was elected leader, sending a full tablet report...
I20260928 16:44:29.828300 1284 ts_tablet_manager.cc:1428] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:44:29.828725 1285 consensus_queue.cc:260] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.829864 981 heartbeater.cc:503] Master 127.30.196.62:46123 was elected leader, sending a full tablet report...
I20260928 16:44:29.830484 1285 ts_tablet_manager.cc:1428] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 16:44:29.833500 1286 raft_consensus.cc:348] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.833822 1286 raft_consensus.cc:374] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.833935 1286 raft_consensus.cc:729] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05885a8db8d4c10948540bd827faa98, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.834470 1286 consensus_queue.cc:260] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.835752 1121 heartbeater.cc:503] Master 127.30.196.62:46123 was elected leader, sending a full tablet report...
I20260928 16:44:29.835877 1286 ts_tablet_manager.cc:1428] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260928 16:44:29.892256 1290 raft_consensus.cc:482] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:29.892477 1290 raft_consensus.cc:504] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.893705 1290 leader_election.cc:288] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:29.898658 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.894114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.30.196.3:46215)
0928 16:44:29.894287 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.894305 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.894428 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.894845 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.894848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.894868 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:29.894992 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.894999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.896004 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.896010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.897284 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.897294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.898107 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.898111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.898241 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.898299 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.898394 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.898496 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":38}
I20260928 16:44:29.898643 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.894213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:46215 (local address 127.30.196.1:38241)
0928 16:44:29.894369 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.894374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.894390 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:29.894644 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.894647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.894704 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.894781 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:29.895140 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.895890 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.897425 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.897964 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.898107 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.898161 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.898415 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.898496 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:29.898650 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.894636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:54161 (local address 127.30.196.2:44371)
0928 16:44:29.894777 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.894781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.894961 (+ 180us) server_negotiation.cc:407] Connection header received
0928 16:44:29.895120 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.895125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.895178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.895278 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:29.895558 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.896148 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.897344 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.897925 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.898069 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.898134 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.898417 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.898509 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:29.898739 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.894391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44371 (local address 127.30.196.3:54161)
0928 16:44:29.894838 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.894853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.894986 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.895285 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.895288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.895308 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:29.895431 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.895437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.896273 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.896280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.897201 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.897211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.898062 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.898066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.898185 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.898297 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.898396 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.898509 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":69,"thread_start_us":145,"threads_started":1}
I20260928 16:44:29.899830 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394729bb8a049cb8481bcade1969b92" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98" is_pre_election: true
I20260928 16:44:29.899830 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394729bb8a049cb8481bcade1969b92" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" is_pre_election: true
I20260928 16:44:29.900209 935 raft_consensus.cc:2454] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:29.900209 1075 raft_consensus.cc:2454] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:29.900767 1149 leader_election.cc:302] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:29.901057 1290 raft_consensus.cc:2781] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:29.901194 1290 raft_consensus.cc:482] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:29.901278 1290 raft_consensus.cc:3037] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:29.902482 1290 raft_consensus.cc:504] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.902844 1290 leader_election.cc:288] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Requested vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:29.903491 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394729bb8a049cb8481bcade1969b92" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98"
I20260928 16:44:29.903491 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7394729bb8a049cb8481bcade1969b92" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
I20260928 16:44:29.903648 1075 raft_consensus.cc:3037] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:29.903648 935 raft_consensus.cc:3037] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:29.905072 935 raft_consensus.cc:2454] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:29.905072 1075 raft_consensus.cc:2454] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:29.905472 1149 leader_election.cc:302] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, 2ef4341ff6fd499f8e529d61e5c7ec0e; no voters:
I20260928 16:44:29.905690 1290 raft_consensus.cc:2781] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:29.905972 1290 raft_consensus.cc:686] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Becoming Leader. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Running, Role: LEADER
I20260928 16:44:29.906399 1290 consensus_queue.cc:237] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } }
I20260928 16:44:29.912946 776 catalog_manager.cc:5495] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "0196bf5438a84233bca0f7d7cbcc9914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:44:29 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613868, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:29.972391 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.961967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.30.196.62:46123)
0928 16:44:29.962463 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.962468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.962494 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:44:29.962562 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.962565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.962623 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.962735 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.963789 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.964460 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.965454 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.965679 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.965773 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.968816 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:29.968819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:29.968821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:29.968843 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:29.970400 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.970921 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.970923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.970925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.970999 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:29.971334 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:29.971337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:29.971338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:29.971537 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:29.971615 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:29.972116 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 16:44:29.972256 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":96}
I20260928 16:44:29.972394 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.961796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46123 (local address 127.0.0.1:60956)
0928 16:44:29.962158 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.962174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.962276 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.962818 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.962821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.963403 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.963540 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.963547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.964607 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.964610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.965308 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.965319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.965927 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.965932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.966054 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.966892 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.966906 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.968638 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:29.970527 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.970538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.970545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.970812 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.971143 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:29.971146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:29.971147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:29.971238 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:29.971729 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:29.971740 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:29.971867 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 16:44:29.972030 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 16:44:29.972191 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
W20260928 16:44:29.973399 982 tablet.cc:2369] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:29.977779 776 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:60956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:44:29.978327 1262 tablet.cc:2369] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:29.978705 776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 16:44:29.979272 1122 tablet.cc:2369] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:29.986529 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 007889d6f90e46f3ab8f29cff27f80d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5951e320b1cd4fb7a20c75a59b5cc965]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:29.986529 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 007889d6f90e46f3ab8f29cff27f80d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5951e320b1cd4fb7a20c75a59b5cc965]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:29.986532 915 tablet_service.cc:1467] Processing CreateTablet for tablet 007889d6f90e46f3ab8f29cff27f80d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5951e320b1cd4fb7a20c75a59b5cc965]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:29.987339 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007889d6f90e46f3ab8f29cff27f80d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.987339 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007889d6f90e46f3ab8f29cff27f80d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.987375 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007889d6f90e46f3ab8f29cff27f80d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:29.990764 1285 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Bootstrap starting.
I20260928 16:44:29.990934 1286 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: Bootstrap starting.
I20260928 16:44:29.990926 1284 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: Bootstrap starting.
I20260928 16:44:29.992146 1285 tablet_bootstrap.cc:654] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.992204 1284 tablet_bootstrap.cc:654] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.992516 1286 tablet_bootstrap.cc:654] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:29.993680 1285 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: No bootstrap required, opened a new log
I20260928 16:44:29.993795 1285 ts_tablet_manager.cc:1397] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:29.994153 1284 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: No bootstrap required, opened a new log
I20260928 16:44:29.994181 1286 tablet_bootstrap.cc:492] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: No bootstrap required, opened a new log
I20260928 16:44:29.994251 1284 ts_tablet_manager.cc:1397] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:29.994314 1286 ts_tablet_manager.cc:1397] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:29.994367 1285 raft_consensus.cc:348] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.994537 1285 raft_consensus.cc:374] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.994578 1285 raft_consensus.cc:729] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.994728 1285 consensus_queue.cc:260] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.994794 1284 raft_consensus.cc:348] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.994915 1284 raft_consensus.cc:374] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.994951 1284 raft_consensus.cc:729] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.995045 1291 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:29.995209 1291 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:29.995164 1284 consensus_queue.cc:260] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.995280 1286 raft_consensus.cc:348] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.995496 1286 raft_consensus.cc:374] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:29.995530 1284 ts_tablet_manager.cc:1428] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:44:29.995543 1286 raft_consensus.cc:729] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05885a8db8d4c10948540bd827faa98, State: Initialized, Role: FOLLOWER
I20260928 16:44:29.995896 1286 consensus_queue.cc:260] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:29.996057 1290 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:29.996201 1290 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:29.996268 1286 ts_tablet_manager.cc:1428] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:29.996559 1292 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:29.996690 1292 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:29.997320 1285 ts_tablet_manager.cc:1428] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:30.260052 1291 raft_consensus.cc:482] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:30.260262 1291 raft_consensus.cc:504] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.261646 1291 leader_election.cc:288] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371), 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167)
I20260928 16:44:30.267287 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.262162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:52163 (local address 127.30.196.3:33167)
0928 16:44:30.262325 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.262331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.262867 (+ 536us) server_negotiation.cc:407] Connection header received
0928 16:44:30.262954 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.262961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.263167 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.263259 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:30.263845 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.264676 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266000 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.266637 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266773 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.266830 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.267044 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.267148 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:30.267309 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.262157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33167 (local address 127.30.196.1:52163)
0928 16:44:30.262560 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.262572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.262659 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.263302 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.263407 (+ 105us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.263425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:30.263660 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.263684 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.264852 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.264854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.265865 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.265875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.266811 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.266899 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.266928 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:44:30.267014 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:30.267082 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":63,"thread_start_us":128,"threads_started":1}
I20260928 16:44:30.267589 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.262215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:59209 (local address 127.30.196.2:44371)
0928 16:44:30.262605 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.262610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.262693 (+ 83us) server_negotiation.cc:407] Connection header received
0928 16:44:30.262808 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.262822 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.262909 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.263178 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:30.263845 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.264710 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266054 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.266644 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266773 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.266830 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.267175 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.267271 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 16:44:30.267321 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.262142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44371 (local address 127.30.196.1:59209)
0928 16:44:30.262470 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.262486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.262652 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.263283 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.263288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.263422 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:30.263631 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.263641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.264883 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.264893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.265853 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.265863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.266912 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.266915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.267011 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.267037 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:44:30.267115 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:44:30.267181 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":89,"thread_start_us":118,"threads_started":1}
I20260928 16:44:30.268077 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007889d6f90e46f3ab8f29cff27f80d1" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98" is_pre_election: true
I20260928 16:44:30.268088 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007889d6f90e46f3ab8f29cff27f80d1" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196bf5438a84233bca0f7d7cbcc9914" is_pre_election: true
I20260928 16:44:30.268257 1075 raft_consensus.cc:2454] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 0.
I20260928 16:44:30.268311 1215 raft_consensus.cc:2454] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 0.
I20260928 16:44:30.268658 869 leader_election.cc:302] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef4341ff6fd499f8e529d61e5c7ec0e, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:30.269346 1291 raft_consensus.cc:2781] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:30.269420 1291 raft_consensus.cc:482] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:30.269448 1291 raft_consensus.cc:3037] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.270582 1291 raft_consensus.cc:504] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.270973 1291 leader_election.cc:288] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 election: Requested vote from peers f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371), 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167)
I20260928 16:44:30.271430 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007889d6f90e46f3ab8f29cff27f80d1" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196bf5438a84233bca0f7d7cbcc9914"
I20260928 16:44:30.271430 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007889d6f90e46f3ab8f29cff27f80d1" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98"
I20260928 16:44:30.271567 1075 raft_consensus.cc:3037] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.271567 1215 raft_consensus.cc:3037] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.272825 1215 raft_consensus.cc:2454] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 1.
I20260928 16:44:30.272832 1075 raft_consensus.cc:2454] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 1.
I20260928 16:44:30.273233 869 leader_election.cc:302] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef4341ff6fd499f8e529d61e5c7ec0e, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:30.273439 1291 raft_consensus.cc:2781] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:30.273689 1291 raft_consensus.cc:686] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 LEADER]: Becoming Leader. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Running, Role: LEADER
I20260928 16:44:30.274085 1291 consensus_queue.cc:237] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.276719 1307 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: TxnStatusTablet state changed. Reason: New leader 2ef4341ff6fd499f8e529d61e5c7ec0e. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.276846 1307 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:30.277161 1309 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:44:30.277402 1309 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:44:30.277858 777 catalog_manager.cc:5495] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1). New cstate: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:30.301237 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.296288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.0.0.1:55512)
0928 16:44:30.296462 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.296482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.296611 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.297023 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.297025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.297044 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:30.297152 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.297157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.298246 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.298249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.299052 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.299063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.299795 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.299799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.299869 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.299923 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.300867 (+ 944us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.300883 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:44:30.300964 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:44:30.301060 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 16:44:30.301234 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.296442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.30.196.1:38241)
0928 16:44:30.296590 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.296595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.296612 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:30.296778 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.296781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.296852 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.296963 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:30.297270 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.298099 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.299173 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.299588 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.299715 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.300148 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.300735 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.300849 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.300998 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.301064 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:30.306972 1075 raft_consensus.cc:1261] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Refusing update from remote peer 2ef4341ff6fd499f8e529d61e5c7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.306969 1215 raft_consensus.cc:1261] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Refusing update from remote peer 2ef4341ff6fd499f8e529d61e5c7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.307708 1307 consensus_queue.cc:1035] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.307927 1291 consensus_queue.cc:1035] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.312230 1292 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: TxnStatusTablet state changed. Reason: New leader 2ef4341ff6fd499f8e529d61e5c7ec0e. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.312252 1290 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: TxnStatusTablet state changed. Reason: New leader 2ef4341ff6fd499f8e529d61e5c7ec0e. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.312422 1292 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:30.312486 1290 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:30.315624 1307 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.315809 1307 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:30.316306 1292 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.316433 1292 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P f05885a8db8d4c10948540bd827faa98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:30.316847 1313 mvcc.cc:205] Tried to move back new op lower bound from 7334354412769755136 to 7334354412645875712. Current Snapshot: MvccSnapshot[applied={T|T < 7334354412769755136}]
I20260928 16:44:30.318710 1307 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.318835 1307 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 2ef4341ff6fd499f8e529d61e5c7ec0e: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:30.320926 1290 tablet_replica.cc:439] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } } }
I20260928 16:44:30.321070 1290 tablet_replica.cc:442] T 007889d6f90e46f3ab8f29cff27f80d1 P 0196bf5438a84233bca0f7d7cbcc9914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:30.322755 1314 mvcc.cc:205] Tried to move back new op lower bound from 7334354412769755136 to 7334354412645875712. Current Snapshot: MvccSnapshot[applied={T|T < 7334354412769755136}]
I20260928 16:44:30.334442 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.329773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.30.196.3:33167)
0928 16:44:30.329926 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.329931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.330129 (+ 198us) server_negotiation.cc:407] Connection header received
0928 16:44:30.330269 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.330272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.330343 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.330462 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:30.330764 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.331549 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.332660 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.333023 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.333144 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.333457 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.334025 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.334144 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.334239 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.334297 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:30.346688 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.343037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.30.196.3:54635)
0928 16:44:30.343402 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.343414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.343579 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.343874 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.343877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.343884 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:30.344011 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.344017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.344755 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.344760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.345386 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.345394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.345885 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.345891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.345979 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.346023 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.346357 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.346365 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:44:30.346454 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:44:30.346534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":109,"threads_started":1}
I20260928 16:44:30.346674 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.343090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:54635 (local address 127.30.196.1:38241)
0928 16:44:30.343242 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.343248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.343523 (+ 275us) server_negotiation.cc:407] Connection header received
0928 16:44:30.343717 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.343720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.343772 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.343861 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:30.344122 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.344639 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.345523 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.345743 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.345857 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.346199 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.346268 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.346350 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.346476 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.346532 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:30.354269 1075 raft_consensus.cc:1261] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.354550 935 raft_consensus.cc:1261] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.354780 1290 consensus_queue.cc:1035] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.355053 1294 consensus_queue.cc:1035] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.380812 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.375978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:52507 (local address 127.30.196.3:33167)
0928 16:44:30.376116 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.376127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.376424 (+ 297us) server_negotiation.cc:407] Connection header received
0928 16:44:30.376594 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.376597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.376674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.376755 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:30.377365 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.378159 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.379223 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.379552 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.379698 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.380225 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.380309 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.380400 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:30.380637 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 16:44:30.380703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:30.381210 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.375879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33167 (local address 127.30.196.1:52507)
0928 16:44:30.376338 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.376351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.376488 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.376780 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.376786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.377095 (+ 309us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:30.377250 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.377257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.378338 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.378341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.379029 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.379039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.379755 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.379759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.379888 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.379962 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:30.380517 (+ 555us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:30.380525 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:44:30.381003 (+ 478us) client_negotiation.cc:240] Negotiation successful
0928 16:44:30.381105 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":236,"threads_started":1}
I20260928 16:44:30.412370 776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:30.412577 776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:30.415515 776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Sending DeleteTablet for 3 replicas of tablet 7394729bb8a049cb8481bcade1969b92
I20260928 16:44:30.416561 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394729bb8a049cb8481bcade1969b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:49122
I20260928 16:44:30.417081 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394729bb8a049cb8481bcade1969b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:53370
I20260928 16:44:30.417500 915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7394729bb8a049cb8481bcade1969b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:55510
I20260928 16:44:30.417546 776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:44:30.417968 1334 tablet_replica.cc:330] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: stopping tablet replica
W20260928 16:44:30.418007 776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:30.418290 1334 raft_consensus.cc:2229] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:30.418653 1334 raft_consensus.cc:2258] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:30.418995 1335 tablet_replica.cc:330] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: stopping tablet replica
I20260928 16:44:30.419275 1335 raft_consensus.cc:2229] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:30.419422 1335 raft_consensus.cc:2258] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:30.420159 1333 tablet_replica.cc:330] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: stopping tablet replica
I20260928 16:44:30.420469 1333 raft_consensus.cc:2229] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:30.420717 1333 raft_consensus.cc:2258] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:30.420790 1335 ts_tablet_manager.cc:1905] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:30.424052 1335 ts_tablet_manager.cc:1918] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:30.424196 1335 log.cc:1196] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/7394729bb8a049cb8481bcade1969b92
I20260928 16:44:30.425002 1335 ts_tablet_manager.cc:1939] T 7394729bb8a049cb8481bcade1969b92 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting consensus metadata
I20260928 16:44:30.425519 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (DEFAULT_TABLE table=test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:30.425837 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:30.425995 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (DEFAULT_TABLE table=test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:30.426316 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:30.428428 1284 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Bootstrap starting.
I20260928 16:44:30.429486 1284 tablet_bootstrap.cc:654] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:30.429874 1286 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Bootstrap starting.
I20260928 16:44:30.425678 915 tablet_service.cc:1467] Processing CreateTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (DEFAULT_TABLE table=test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:30.430583 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:30.430847 1286 tablet_bootstrap.cc:654] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:30.433084 1285 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Bootstrap starting.
I20260928 16:44:30.433548 1334 ts_tablet_manager.cc:1905] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:30.434036 1285 tablet_bootstrap.cc:654] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:30.436062 765 catalog_manager.cc:4841] TS 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241): tablet 7394729bb8a049cb8481bcade1969b92 (table test-table [id=79f3f3f731154842bebc91210a29779a]) successfully deleted
I20260928 16:44:30.436344 1333 ts_tablet_manager.cc:1905] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:30.438638 1334 ts_tablet_manager.cc:1918] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:30.438745 1334 log.cc:1196] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/7394729bb8a049cb8481bcade1969b92
I20260928 16:44:30.439118 1334 ts_tablet_manager.cc:1939] T 7394729bb8a049cb8481bcade1969b92 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting consensus metadata
I20260928 16:44:30.440359 1333 ts_tablet_manager.cc:1918] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:30.440465 1333 log.cc:1196] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/7394729bb8a049cb8481bcade1969b92
I20260928 16:44:30.440797 1333 ts_tablet_manager.cc:1939] T 7394729bb8a049cb8481bcade1969b92 P f05885a8db8d4c10948540bd827faa98: Deleting consensus metadata
I20260928 16:44:30.441247 762 catalog_manager.cc:4841] TS 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167): tablet 7394729bb8a049cb8481bcade1969b92 (table test-table [id=79f3f3f731154842bebc91210a29779a]) successfully deleted
I20260928 16:44:30.441973 1284 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: No bootstrap required, opened a new log
I20260928 16:44:30.442071 1284 ts_tablet_manager.cc:1397] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260928 16:44:30.442556 1284 raft_consensus.cc:348] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.442684 1284 raft_consensus.cc:374] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:30.442724 1284 raft_consensus.cc:729] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Initialized, Role: FOLLOWER
I20260928 16:44:30.442844 1284 consensus_queue.cc:260] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.442858 765 catalog_manager.cc:4841] TS f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371): tablet 7394729bb8a049cb8481bcade1969b92 (table test-table [id=79f3f3f731154842bebc91210a29779a]) successfully deleted
I20260928 16:44:30.443111 1284 ts_tablet_manager.cc:1428] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:30.444118 1286 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: No bootstrap required, opened a new log
I20260928 16:44:30.444202 1286 ts_tablet_manager.cc:1397] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260928 16:44:30.444648 1286 raft_consensus.cc:348] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.444747 1286 raft_consensus.cc:374] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:30.444785 1286 raft_consensus.cc:729] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05885a8db8d4c10948540bd827faa98, State: Initialized, Role: FOLLOWER
I20260928 16:44:30.444897 1286 consensus_queue.cc:260] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.445138 1286 ts_tablet_manager.cc:1428] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:30.446151 1285 tablet_bootstrap.cc:492] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: No bootstrap required, opened a new log
I20260928 16:44:30.446245 1285 ts_tablet_manager.cc:1397] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 16:44:30.446678 1285 raft_consensus.cc:348] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.446781 1285 raft_consensus.cc:374] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:30.446810 1285 raft_consensus.cc:729] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Initialized, Role: FOLLOWER
I20260928 16:44:30.446924 1285 consensus_queue.cc:260] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.447162 1285 ts_tablet_manager.cc:1428] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:30.459692 1330 raft_consensus.cc:482] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:30.459882 1330 raft_consensus.cc:504] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.460346 1330 leader_election.cc:288] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:30.461005 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670b3bb7f53b49aaafdb9e5e9eb87b9c" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" is_pre_election: true
I20260928 16:44:30.461161 935 raft_consensus.cc:2454] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:30.461486 1149 leader_election.cc:302] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, 2ef4341ff6fd499f8e529d61e5c7ec0e; no voters:
I20260928 16:44:30.461505 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670b3bb7f53b49aaafdb9e5e9eb87b9c" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98" is_pre_election: true
I20260928 16:44:30.461663 1075 raft_consensus.cc:2454] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:30.461680 1330 raft_consensus.cc:2781] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:30.461725 1330 raft_consensus.cc:482] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:30.461761 1330 raft_consensus.cc:3037] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.463210 1330 raft_consensus.cc:504] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.463555 1330 leader_election.cc:288] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Requested vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:30.463912 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670b3bb7f53b49aaafdb9e5e9eb87b9c" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
I20260928 16:44:30.464018 935 raft_consensus.cc:3037] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.463997 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670b3bb7f53b49aaafdb9e5e9eb87b9c" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98"
I20260928 16:44:30.464110 1075 raft_consensus.cc:3037] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:30.465101 935 raft_consensus.cc:2454] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:30.465238 1075 raft_consensus.cc:2454] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:30.465411 1149 leader_election.cc:302] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, 2ef4341ff6fd499f8e529d61e5c7ec0e; no voters:
I20260928 16:44:30.465663 1330 raft_consensus.cc:2781] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:30.465760 1330 raft_consensus.cc:686] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Becoming Leader. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Running, Role: LEADER
I20260928 16:44:30.465879 1330 consensus_queue.cc:237] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:30.467516 776 catalog_manager.cc:5495] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "0196bf5438a84233bca0f7d7cbcc9914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } health_report { overall_health: HEALTHY } } }
I20260928 16:44:30.499910 1075 raft_consensus.cc:1261] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.499910 935 raft_consensus.cc:1261] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:30.500375 1330 consensus_queue.cc:1035] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.500600 1294 consensus_queue.cc:1035] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:30.503785 1312 mvcc.cc:205] Tried to move back new op lower bound from 7334354413563695104 to 7334354413429530624. Current Snapshot: MvccSnapshot[applied={T|T < 7334354413563695104}]
W20260928 16:44:30.515889 1314 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:44:32.073685 774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:32.073872 774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:32.076664 774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Sending DeleteTablet for 3 replicas of tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c
I20260928 16:44:32.077437 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:53370
I20260928 16:44:32.077450 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:49122
I20260928 16:44:32.077450 915 tablet_service.cc:1514] Processing DeleteTablet for tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:55510
I20260928 16:44:32.077795 1365 tablet_replica.cc:330] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: stopping tablet replica
I20260928 16:44:32.077828 1367 tablet_replica.cc:330] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: stopping tablet replica
I20260928 16:44:32.077956 1365 raft_consensus.cc:2229] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:32.077970 1367 raft_consensus.cc:2229] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:32.078095 1367 raft_consensus.cc:2258] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:32.078192 1365 raft_consensus.cc:2258] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:32.078799 1365 ts_tablet_manager.cc:1905] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.079865 1366 tablet_replica.cc:330] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: stopping tablet replica
I20260928 16:44:32.080081 1366 raft_consensus.cc:2229] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:32.080291 1366 raft_consensus.cc:2258] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:32.081081 1366 ts_tablet_manager.cc:1905] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.081691 1365 ts_tablet_manager.cc:1918] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:44:32.081784 1365 log.cc:1196] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/670b3bb7f53b49aaafdb9e5e9eb87b9c
I20260928 16:44:32.082072 1365 ts_tablet_manager.cc:1939] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 0196bf5438a84233bca0f7d7cbcc9914: Deleting consensus metadata
I20260928 16:44:32.082780 762 catalog_manager.cc:4841] TS 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167): tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (table test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]) successfully deleted
I20260928 16:44:32.083022 1367 ts_tablet_manager.cc:1905] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.084080 1366 ts_tablet_manager.cc:1918] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:44:32.084223 1366 log.cc:1196] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/670b3bb7f53b49aaafdb9e5e9eb87b9c
I20260928 16:44:32.084599 1366 ts_tablet_manager.cc:1939] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P f05885a8db8d4c10948540bd827faa98: Deleting consensus metadata
I20260928 16:44:32.085908 1367 ts_tablet_manager.cc:1918] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:44:32.086002 1367 log.cc:1196] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/670b3bb7f53b49aaafdb9e5e9eb87b9c
I20260928 16:44:32.086272 1367 ts_tablet_manager.cc:1939] T 670b3bb7f53b49aaafdb9e5e9eb87b9c P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting consensus metadata
I20260928 16:44:32.086597 765 catalog_manager.cc:4841] TS f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371): tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (table test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]) successfully deleted
I20260928 16:44:32.087127 765 catalog_manager.cc:4841] TS 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241): tablet 670b3bb7f53b49aaafdb9e5e9eb87b9c (table test-table [id=cbc84d55585c4fbfa34d36ffd7621b55]) successfully deleted
Sep 28 16:44:32 dist-test-slave-5rg1 krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:44:32 dist-test-slave-5rg1 krb5kdc[733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:32.107882 1375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.098322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.30.196.62:46123)
0928 16:44:32.098666 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.098674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.098688 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:32.098778 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.098781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.098825 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.098905 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:32.099770 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.100350 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.101241 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.101511 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.101605 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.104316 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:32.104319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:32.104322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:32.104348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:32.105926 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:32.106338 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:32.106341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:32.106343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:32.106428 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:32.106718 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:32.106721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:32.106722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:32.106992 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:32.107112 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:32.107421 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:44:32.107580 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":150,"threads_started":1}
I20260928 16:44:32.111029 774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:32.111454 774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:32.116331 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 9e899c4369ce41fbae785efd920b0f6e (DEFAULT_TABLE table=test-table [id=90d30e97e4484a2a923f52433f97c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.116291 915 tablet_service.cc:1467] Processing CreateTablet for tablet 9e899c4369ce41fbae785efd920b0f6e (DEFAULT_TABLE table=test-table [id=90d30e97e4484a2a923f52433f97c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.116560 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 9e899c4369ce41fbae785efd920b0f6e (DEFAULT_TABLE table=test-table [id=90d30e97e4484a2a923f52433f97c500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.116648 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e899c4369ce41fbae785efd920b0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.116658 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e899c4369ce41fbae785efd920b0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.116809 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e899c4369ce41fbae785efd920b0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.118669 1376 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Bootstrap starting.
I20260928 16:44:32.119427 1378 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Bootstrap starting.
I20260928 16:44:32.119524 1377 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Bootstrap starting.
I20260928 16:44:32.119671 1376 tablet_bootstrap.cc:654] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.120540 1378 tablet_bootstrap.cc:654] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.120555 1377 tablet_bootstrap.cc:654] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.121203 1376 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: No bootstrap required, opened a new log
I20260928 16:44:32.121325 1376 ts_tablet_manager.cc:1397] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:44:32.121850 1377 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: No bootstrap required, opened a new log
I20260928 16:44:32.121811 1376 raft_consensus.cc:348] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.121877 1378 tablet_bootstrap.cc:492] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: No bootstrap required, opened a new log
I20260928 16:44:32.121922 1376 raft_consensus.cc:374] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.121942 1377 ts_tablet_manager.cc:1397] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:32.121978 1376 raft_consensus.cc:729] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05885a8db8d4c10948540bd827faa98, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.121984 1378 ts_tablet_manager.cc:1397] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:32.122105 1376 consensus_queue.cc:260] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.122354 1377 raft_consensus.cc:348] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.122478 1377 raft_consensus.cc:374] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.122488 1376 ts_tablet_manager.cc:1428] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 16:44:32.122516 1377 raft_consensus.cc:729] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.122639 1377 consensus_queue.cc:260] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.122928 1377 ts_tablet_manager.cc:1428] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:32.123263 1378 raft_consensus.cc:348] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.123409 1378 raft_consensus.cc:374] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.123473 1378 raft_consensus.cc:729] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.123648 1378 consensus_queue.cc:260] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.124044 1378 ts_tablet_manager.cc:1428] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 16:44:32.410879 1360 raft_consensus.cc:482] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:32.411130 1360 raft_consensus.cc:504] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.411630 1360 leader_election.cc:288] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:32.412375 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e899c4369ce41fbae785efd920b0f6e" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" is_pre_election: true
I20260928 16:44:32.412375 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e899c4369ce41fbae785efd920b0f6e" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98" is_pre_election: true
I20260928 16:44:32.412593 1075 raft_consensus.cc:2454] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:32.412593 935 raft_consensus.cc:2454] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 0.
I20260928 16:44:32.412976 1149 leader_election.cc:302] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, 2ef4341ff6fd499f8e529d61e5c7ec0e; no voters:
I20260928 16:44:32.413214 1360 raft_consensus.cc:2781] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:32.413292 1360 raft_consensus.cc:482] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:32.413347 1360 raft_consensus.cc:3037] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.414513 1360 raft_consensus.cc:504] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.414896 1360 leader_election.cc:288] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Requested vote from peers 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241), f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371)
I20260928 16:44:32.415467 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e899c4369ce41fbae785efd920b0f6e" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e"
I20260928 16:44:32.415485 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e899c4369ce41fbae785efd920b0f6e" candidate_uuid: "0196bf5438a84233bca0f7d7cbcc9914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98"
I20260928 16:44:32.415623 935 raft_consensus.cc:3037] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.415623 1075 raft_consensus.cc:3037] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.416855 935 raft_consensus.cc:2454] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:32.416855 1075 raft_consensus.cc:2454] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196bf5438a84233bca0f7d7cbcc9914 in term 1.
I20260928 16:44:32.417224 1149 leader_election.cc:302] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:32.417520 1360 raft_consensus.cc:2781] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:32.417661 1360 raft_consensus.cc:686] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Becoming Leader. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Running, Role: LEADER
I20260928 16:44:32.417829 1360 consensus_queue.cc:237] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.420044 774 catalog_manager.cc:5495] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "0196bf5438a84233bca0f7d7cbcc9914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } health_report { overall_health: HEALTHY } } }
I20260928 16:44:32.502665 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.498832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.30.196.3:33167)
0928 16:44:32.499104 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.499113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.499127 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:32.499214 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.499221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.499263 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.499353 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:32.499615 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.500265 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.501499 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.501738 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.501821 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.502072 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:32.502142 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:32.502201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:32.502332 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:44:32.502375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260928 16:44:32.509851 1075 raft_consensus.cc:1261] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:32.509851 935 raft_consensus.cc:1261] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Refusing update from remote peer 0196bf5438a84233bca0f7d7cbcc9914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:32.510365 1356 consensus_queue.cc:1035] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:32.510545 1360 consensus_queue.cc:1035] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:32.547541 774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:32.547744 774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:32.549731 774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Sending DeleteTablet for 3 replicas of tablet 9e899c4369ce41fbae785efd920b0f6e
I20260928 16:44:32.550321 915 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e899c4369ce41fbae785efd920b0f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:55510
I20260928 16:44:32.550318 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e899c4369ce41fbae785efd920b0f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:49122
I20260928 16:44:32.550521 1366 tablet_replica.cc:330] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: stopping tablet replica
I20260928 16:44:32.550522 1367 tablet_replica.cc:330] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: stopping tablet replica
I20260928 16:44:32.550647 1367 raft_consensus.cc:2229] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:32.550645 1366 raft_consensus.cc:2229] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:32.550760 1367 raft_consensus.cc:2258] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:32.550758 1366 raft_consensus.cc:2258] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:32.550925 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e899c4369ce41fbae785efd920b0f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:32 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:53370
I20260928 16:44:32.551087 1365 tablet_replica.cc:330] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: stopping tablet replica
I20260928 16:44:32.551191 1365 raft_consensus.cc:2229] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:32.551146 774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:44:32.551388 1365 raft_consensus.cc:2258] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:44:32.551515 774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:32.552186 1366 ts_tablet_manager.cc:1905] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.552719 1367 ts_tablet_manager.cc:1905] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.554785 1365 ts_tablet_manager.cc:1905] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:32.554785 1366 ts_tablet_manager.cc:1918] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:32.554862 1366 log.cc:1196] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/9e899c4369ce41fbae785efd920b0f6e
I20260928 16:44:32.555289 1366 ts_tablet_manager.cc:1939] T 9e899c4369ce41fbae785efd920b0f6e P f05885a8db8d4c10948540bd827faa98: Deleting consensus metadata
I20260928 16:44:32.555464 1367 ts_tablet_manager.cc:1918] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:32.555533 1367 log.cc:1196] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/9e899c4369ce41fbae785efd920b0f6e
I20260928 16:44:32.555819 1367 ts_tablet_manager.cc:1939] T 9e899c4369ce41fbae785efd920b0f6e P 2ef4341ff6fd499f8e529d61e5c7ec0e: Deleting consensus metadata
I20260928 16:44:32.556226 765 catalog_manager.cc:4841] TS f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371): tablet 9e899c4369ce41fbae785efd920b0f6e (table test-table [id=90d30e97e4484a2a923f52433f97c500]) successfully deleted
I20260928 16:44:32.556695 765 catalog_manager.cc:4841] TS 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241): tablet 9e899c4369ce41fbae785efd920b0f6e (table test-table [id=90d30e97e4484a2a923f52433f97c500]) successfully deleted
I20260928 16:44:32.556799 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 0f212998259b4c9297cf738b27f517a8 (DEFAULT_TABLE table=test-table [id=8b0968babf0f4da8adadbfabc91ec78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.557049 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f212998259b4c9297cf738b27f517a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.557145 915 tablet_service.cc:1467] Processing CreateTablet for tablet 0f212998259b4c9297cf738b27f517a8 (DEFAULT_TABLE table=test-table [id=8b0968babf0f4da8adadbfabc91ec78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.557381 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f212998259b4c9297cf738b27f517a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.557387 1365 ts_tablet_manager.cc:1918] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:32.557451 1365 log.cc:1196] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/9e899c4369ce41fbae785efd920b0f6e
I20260928 16:44:32.557701 1365 ts_tablet_manager.cc:1939] T 9e899c4369ce41fbae785efd920b0f6e P 0196bf5438a84233bca0f7d7cbcc9914: Deleting consensus metadata
I20260928 16:44:32.558358 762 catalog_manager.cc:4841] TS 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167): tablet 9e899c4369ce41fbae785efd920b0f6e (table test-table [id=90d30e97e4484a2a923f52433f97c500]) successfully deleted
I20260928 16:44:32.559124 1376 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Bootstrap starting.
I20260928 16:44:32.559252 1378 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Bootstrap starting.
I20260928 16:44:32.559326 1195 tablet_service.cc:1467] Processing CreateTablet for tablet 0f212998259b4c9297cf738b27f517a8 (DEFAULT_TABLE table=test-table [id=8b0968babf0f4da8adadbfabc91ec78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:32.559569 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f212998259b4c9297cf738b27f517a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:32.560146 1378 tablet_bootstrap.cc:654] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.560146 1376 tablet_bootstrap.cc:654] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.561373 1378 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e: No bootstrap required, opened a new log
I20260928 16:44:32.561458 1378 ts_tablet_manager.cc:1397] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:32.561609 1376 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: No bootstrap required, opened a new log
I20260928 16:44:32.561681 1376 ts_tablet_manager.cc:1397] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:32.561717 1377 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Bootstrap starting.
I20260928 16:44:32.561894 1378 raft_consensus.cc:348] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.562001 1378 raft_consensus.cc:374] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.562034 1378 raft_consensus.cc:729] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.562139 1376 raft_consensus.cc:348] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.562176 1378 consensus_queue.cc:260] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.562233 1376 raft_consensus.cc:374] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.562265 1376 raft_consensus.cc:729] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05885a8db8d4c10948540bd827faa98, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.562381 1376 consensus_queue.cc:260] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.562453 1378 ts_tablet_manager.cc:1428] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:44:32.562495 1377 tablet_bootstrap.cc:654] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:32.562634 1376 ts_tablet_manager.cc:1428] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:32.563582 1377 tablet_bootstrap.cc:492] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: No bootstrap required, opened a new log
I20260928 16:44:32.563702 1377 ts_tablet_manager.cc:1397] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:32.564049 1377 raft_consensus.cc:348] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.564162 1377 raft_consensus.cc:374] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:32.564201 1377 raft_consensus.cc:729] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196bf5438a84233bca0f7d7cbcc9914, State: Initialized, Role: FOLLOWER
I20260928 16:44:32.564359 1377 consensus_queue.cc:260] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.564657 1377 ts_tablet_manager.cc:1428] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:44:32.699193 1319 raft_consensus.cc:482] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:32.699357 1319 raft_consensus.cc:504] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.699873 1319 leader_election.cc:288] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371), 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167)
I20260928 16:44:32.700438 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f212998259b4c9297cf738b27f517a8" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196bf5438a84233bca0f7d7cbcc9914" is_pre_election: true
I20260928 16:44:32.700467 1074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f212998259b4c9297cf738b27f517a8" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98" is_pre_election: true
I20260928 16:44:32.700609 1215 raft_consensus.cc:2454] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 0.
I20260928 16:44:32.700595 1074 raft_consensus.cc:2454] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 0.
I20260928 16:44:32.700948 869 leader_election.cc:302] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef4341ff6fd499f8e529d61e5c7ec0e, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:32.701197 1319 raft_consensus.cc:2781] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:32.701253 1319 raft_consensus.cc:482] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:32.701282 1319 raft_consensus.cc:3037] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.702342 1319 raft_consensus.cc:504] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.702714 1319 leader_election.cc:288] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 election: Requested vote from peers f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371), 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167)
I20260928 16:44:32.703215 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f212998259b4c9297cf738b27f517a8" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196bf5438a84233bca0f7d7cbcc9914"
I20260928 16:44:32.703215 1074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f212998259b4c9297cf738b27f517a8" candidate_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f05885a8db8d4c10948540bd827faa98"
I20260928 16:44:32.703351 1215 raft_consensus.cc:3037] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.703351 1074 raft_consensus.cc:3037] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:32.704618 1215 raft_consensus.cc:2454] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 1.
I20260928 16:44:32.704618 1074 raft_consensus.cc:2454] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef4341ff6fd499f8e529d61e5c7ec0e in term 1.
I20260928 16:44:32.704972 869 leader_election.cc:302] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0196bf5438a84233bca0f7d7cbcc9914, 2ef4341ff6fd499f8e529d61e5c7ec0e, f05885a8db8d4c10948540bd827faa98; no voters:
I20260928 16:44:32.705237 1319 raft_consensus.cc:2781] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:32.705353 1319 raft_consensus.cc:686] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [term 1 LEADER]: Becoming Leader. State: Replica: 2ef4341ff6fd499f8e529d61e5c7ec0e, State: Running, Role: LEADER
I20260928 16:44:32.705509 1319 consensus_queue.cc:237] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } }
I20260928 16:44:32.707263 774 catalog_manager.cc:5495] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1). New cstate: current_term: 1 leader_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef4341ff6fd499f8e529d61e5c7ec0e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:32.728526 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.725128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.30.196.1:38241)
0928 16:44:32.725468 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.725478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.725501 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:32.725552 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.725554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.725597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.725706 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:32.726015 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.726558 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.727390 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.727637 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.727790 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.728030 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:32.728110 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:32.728191 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:32.728307 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:44:32.728362 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":146,"threads_started":1}
I20260928 16:44:32.738545 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.734772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.30.196.1:52985)
0928 16:44:32.735064 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:32.735076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:32.735160 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:32.735448 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:32.735450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:32.735458 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:32.735639 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.735649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.736676 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.736679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.737268 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.737277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.737858 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.737861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.737924 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.737951 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:32.738238 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:32.738242 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:44:32.738294 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 16:44:32.738371 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
I20260928 16:44:32.738839 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.734861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:52985 (local address 127.30.196.1:38241)
0928 16:44:32.734982 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.734987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.735147 (+ 160us) server_negotiation.cc:407] Connection header received
0928 16:44:32.735263 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.735266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.735308 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.735385 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:32.735768 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.736548 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.737379 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.737742 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.737838 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.738058 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:32.738152 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:32.738213 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:32.738344 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:44:32.738554 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:44:32.744696 1074 raft_consensus.cc:1261] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Refusing update from remote peer 2ef4341ff6fd499f8e529d61e5c7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:32.744757 1215 raft_consensus.cc:1261] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Refusing update from remote peer 2ef4341ff6fd499f8e529d61e5c7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:32.745133 1319 consensus_queue.cc:1035] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05885a8db8d4c10948540bd827faa98" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:32.745304 1351 consensus_queue.cc:1035] T 0f212998259b4c9297cf738b27f517a8 P 2ef4341ff6fd499f8e529d61e5c7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196bf5438a84233bca0f7d7cbcc9914" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:44:32.758203 1343 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:44:34.317759 774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:34.318022 774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:34.321130 774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ed19598d834f02a94f7ecc173274e7: Sending DeleteTablet for 3 replicas of tablet 0f212998259b4c9297cf738b27f517a8
I20260928 16:44:34.321871 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f212998259b4c9297cf738b27f517a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:53370
I20260928 16:44:34.321873 911 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f212998259b4c9297cf738b27f517a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:55510
I20260928 16:44:34.321973 1055 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f212998259b4c9297cf738b27f517a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:49122
I20260928 16:44:34.322453 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 850
I20260928 16:44:34.323295 1422 tablet_replica.cc:330] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: stopping tablet replica
I20260928 16:44:34.323467 1422 raft_consensus.cc:2229] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:34.323637 1422 raft_consensus.cc:2258] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:34.323956 1421 tablet_replica.cc:330] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: stopping tablet replica
I20260928 16:44:34.324177 1421 raft_consensus.cc:2229] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:34.324376 1421 raft_consensus.cc:2258] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:34.324564 1422 ts_tablet_manager.cc:1905] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:34.325067 1421 ts_tablet_manager.cc:1905] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:34.328811 1422 ts_tablet_manager.cc:1918] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:44:34.328953 1422 log.cc:1196] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/0f212998259b4c9297cf738b27f517a8
I20260928 16:44:34.329035 1421 ts_tablet_manager.cc:1918] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:44:34.329145 1421 log.cc:1196] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/0f212998259b4c9297cf738b27f517a8
I20260928 16:44:34.329258 1422 ts_tablet_manager.cc:1939] T 0f212998259b4c9297cf738b27f517a8 P f05885a8db8d4c10948540bd827faa98: Deleting consensus metadata
I20260928 16:44:34.329474 1421 ts_tablet_manager.cc:1939] T 0f212998259b4c9297cf738b27f517a8 P 0196bf5438a84233bca0f7d7cbcc9914: Deleting consensus metadata
I20260928 16:44:34.330317 762 catalog_manager.cc:4841] TS 0196bf5438a84233bca0f7d7cbcc9914 (127.30.196.3:33167): tablet 0f212998259b4c9297cf738b27f517a8 (table test-table [id=8b0968babf0f4da8adadbfabc91ec78f]) successfully deleted
I20260928 16:44:34.330564 765 catalog_manager.cc:4841] TS f05885a8db8d4c10948540bd827faa98 (127.30.196.2:44371): tablet 0f212998259b4c9297cf738b27f517a8 (table test-table [id=8b0968babf0f4da8adadbfabc91ec78f]) successfully deleted
W20260928 16:44:34.333752 765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:38241 (error 108)
I20260928 16:44:34.334054 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 990
I20260928 16:44:34.335537 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:34.334860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38241 (local address 127.0.0.1:55536)
0928 16:44:34.335377 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:34.335442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":82,"threads_started":1}
W20260928 16:44:34.335697 765 catalog_manager.cc:4584] TS 2ef4341ff6fd499f8e529d61e5c7ec0e (127.30.196.1:38241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f212998259b4c9297cf738b27f517a8: Network error: Client connection negotiation failed: client connection to 127.30.196.1:38241: connect: Connection refused (error 111)
I20260928 16:44:34.340732 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1130
I20260928 16:44:34.347954 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 746
2026-09-28T16:44:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6894 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:34 dist-test-slave-5rg1 krb5kdc[1428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:34 dist-test-slave-5rg1 krb5kdc[1428](info): set up 2 sockets
Sep 28 16:44:34 dist-test-slave-5rg1 krb5kdc[1428](info): commencing operation
krb5kdc: starting...
W20260928 16:44:35.436125 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:35 dist-test-slave-5rg1 krb5kdc[1428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:35.567365 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36663
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:35.681583 1442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:35.681901 1442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:35.681974 1442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:35.686131 1442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:35.686220 1442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:35.686261 1442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:35.686322 1442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:35.686362 1442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:35.691913 1442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:35.693089 1442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:35.694167 1442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:35.703311 1442 server_base.cc:1034] running on GCE node
W20260928 16:44:35.703276 1450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:35.703256 1447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:35.703256 1448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:35.704237 1442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:35.705411 1442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:35.706589 1442 hybrid_clock.cc:648] HybridClock initialized: now 1790613875706571 us; error 36 us; skew 500 ppm
Sep 28 16:44:35 dist-test-slave-5rg1 krb5kdc[1428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:35.709527 1442 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:35.710444 1442 webserver.cc:460] Webserver started at http://127.30.196.62:40057/ using document root <none> and password file <none>
I20260928 16:44:35.710978 1442 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:35.711042 1442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:35.711210 1442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:35.713356 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "3cfcdf552d2878d01fb61c51210a419f"
server_key_iv: "543c8de8cdce41d69c8ab7023424e2dd"
server_key_version: "encryptionkey@0"
I20260928 16:44:35.713883 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "3cfcdf552d2878d01fb61c51210a419f"
server_key_iv: "543c8de8cdce41d69c8ab7023424e2dd"
server_key_version: "encryptionkey@0"
I20260928 16:44:35.717769 1442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:35.720407 1457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:35.721771 1442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:35.722019 1442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "3cfcdf552d2878d01fb61c51210a419f"
server_key_iv: "543c8de8cdce41d69c8ab7023424e2dd"
server_key_version: "encryptionkey@0"
I20260928 16:44:35.722151 1442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:35.740411 1442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:35.743497 1442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:35.743795 1442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:35.751322 1442 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36663
I20260928 16:44:35.751349 1509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36663 every 8 connection(s)
I20260928 16:44:35.752341 1442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:35.754078 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1442
I20260928 16:44:35.754189 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:35.754496 31504 external_mini_cluster.cc:1428] Setting key 16d6f57f070252fa359c367b0b206bb5
I20260928 16:44:35.755584 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:35.761158 1510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e: Bootstrap starting.
Sep 28 16:44:35 dist-test-slave-5rg1 krb5kdc[1428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:35.763654 1510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:35.764539 1510 log.cc:824] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:35.766842 1510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e: No bootstrap required, opened a new log
I20260928 16:44:35.769064 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:35.755962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.30.196.62:36663)
0928 16:44:35.756396 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:35.756410 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:35.756444 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:44:35.757083 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:35.757098 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:35.757406 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:35.757750 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:35.758584 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.759554 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.760442 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.760915 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.761095 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.763874 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:35.763879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:35.763893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:35.763927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:35.765817 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:35.766335 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:35.766341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:35.766347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:35.766482 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:35.766766 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:35.766769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:35.766771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:35.767276 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:35.767462 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:35.767801 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:44:35.768093 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":148,"threads_started":1}
I20260928 16:44:35.771203 1510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } }
I20260928 16:44:35.771478 1510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:35.771555 1510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cd96de13a24fe19f4ee2f70ceb6a6e, State: Initialized, Role: FOLLOWER
I20260928 16:44:35.772029 1510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } }
I20260928 16:44:35.772166 1510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:35.772208 1510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:35.772272 1510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:35.773108 1510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } }
I20260928 16:44:35.773393 1510 leader_election.cc:302] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cd96de13a24fe19f4ee2f70ceb6a6e; no voters:
I20260928 16:44:35.773635 1510 leader_election.cc:288] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:35.773784 1515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:35.774134 1515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [term 1 LEADER]: Becoming Leader. State: Replica: 50cd96de13a24fe19f4ee2f70ceb6a6e, State: Running, Role: LEADER
I20260928 16:44:35.774482 1515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } }
I20260928 16:44:35.774812 1510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:35.776094 1517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cd96de13a24fe19f4ee2f70ceb6a6e. Latest consensus state: current_term: 1 leader_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } } }
I20260928 16:44:35.776226 1517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:35.776603 1524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:35.777273 1516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cd96de13a24fe19f4ee2f70ceb6a6e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36663 } } }
I20260928 16:44:35.777377 1516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:35.779307 1524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:35.784780 1524 catalog_manager.cc:1347] Generated new cluster ID: a02c73826a104e9e98b8a3e1840e2bea
I20260928 16:44:35.784860 1524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:35.800848 1524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:35.802153 1524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:35.811614 1524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cd96de13a24fe19f4ee2f70ceb6a6e: Generated new TSK 0
I20260928 16:44:35.812549 1524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:35.844470 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:35.959444 1536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:35.959786 1536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:35.959833 1536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:35.964553 1536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:35.964612 1536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:35.964705 1536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:35.970041 1536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:35.971177 1536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:35.972261 1536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:35.982656 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:35.982672 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:35.982870 1536 server_base.cc:1034] running on GCE node
W20260928 16:44:35.982779 1544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:35.983445 1536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:35.984198 1536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:35.985399 1536 hybrid_clock.cc:648] HybridClock initialized: now 1790613875985386 us; error 30 us; skew 500 ppm
Sep 28 16:44:35 dist-test-slave-5rg1 krb5kdc[1428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:35.988845 1536 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:35.989787 1536 webserver.cc:460] Webserver started at http://127.30.196.1:40937/ using document root <none> and password file <none>
I20260928 16:44:35.990408 1536 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:35.990509 1536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:35.990756 1536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:35.992982 1536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "9179830ad7bb4eb088939fbc5e2900d0"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "4d0f19c530f4881e0491e992027ed84d"
server_key_iv: "0ab08b5a325e3e8a0be7b87e89ce0f82"
server_key_version: "encryptionkey@0"
I20260928 16:44:35.993701 1536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "9179830ad7bb4eb088939fbc5e2900d0"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "4d0f19c530f4881e0491e992027ed84d"
server_key_iv: "0ab08b5a325e3e8a0be7b87e89ce0f82"
server_key_version: "encryptionkey@0"
I20260928 16:44:35.998046 1536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:44:36.001082 1552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.002503 1536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:36.002682 1536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "9179830ad7bb4eb088939fbc5e2900d0"
format_stamp: "Formatted at 2026-09-28 16:44:35 on dist-test-slave-5rg1"
server_key: "4d0f19c530f4881e0491e992027ed84d"
server_key_iv: "0ab08b5a325e3e8a0be7b87e89ce0f82"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.002820 1536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:36.017925 1536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:36.021461 1536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:36.021694 1536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:36.022378 1536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:36.023388 1536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:36.023478 1536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.023563 1536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:36.023618 1536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.033030 1536 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:38805
I20260928 16:44:36.033056 1665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:38805 every 8 connection(s)
I20260928 16:44:36.034051 1536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:36.041652 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1536
I20260928 16:44:36.041793 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:36.042109 31504 external_mini_cluster.cc:1428] Setting key 672533ef1adea2342ebbc3b82854f267
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613875, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:36.050727 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.036828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:53479 (local address 127.30.196.62:36663)
0928 16:44:36.037019 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.037023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.037912 (+ 889us) server_negotiation.cc:407] Connection header received
0928 16:44:36.039308 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.039311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.039361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.039445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.040919 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.041751 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.043070 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.043317 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.043435 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.046257 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:36.046262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:36.046267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:36.046302 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:36.048238 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.048771 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.048778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.048783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.048878 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.049290 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.049296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.049300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.049608 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:36.049730 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:36.050350 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 16:44:36.050516 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:44:36.051499 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.037209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36663 (local address 127.30.196.1:53479)
0928 16:44:36.037736 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.037784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.039091 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.039612 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.039618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.040121 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.040630 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.040644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.041949 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.041953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.042931 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.042941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.043484 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.043488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.043659 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.044376 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.044395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.046058 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:36.048361 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.048367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.048381 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.048662 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.048993 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.048995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.048997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.049187 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.049757 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:36.049763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:36.050081 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 16:44:36.050259 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 16:44:36.050536 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:36.052886 1666 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36663
I20260928 16:44:36.053200 1666 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:36.053753 1666 heartbeater.cc:511] Master 127.30.196.62:36663 requested a full tablet report, sending...
I20260928 16:44:36.055349 1473 ts_manager.cc:194] Registered new tserver with Master: 9179830ad7bb4eb088939fbc5e2900d0 (127.30.196.1:38805)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:36.056735 1473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:53479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:36.072147 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:36.193427 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:36.193737 1671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:36.193784 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:36.198664 1671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:36.198740 1671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:36.198843 1671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:36.204615 1671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:36.205853 1671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:36.207193 1671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:36.217734 1676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:36.217715 1679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:36.217718 1677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:36.218204 1671 server_base.cc:1034] running on GCE node
I20260928 16:44:36.218639 1671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:36.219419 1671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:36.220613 1671 hybrid_clock.cc:648] HybridClock initialized: now 1790613876220584 us; error 48 us; skew 500 ppm
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:36.223596 1671 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:36.224608 1671 webserver.cc:460] Webserver started at http://127.30.196.2:38449/ using document root <none> and password file <none>
I20260928 16:44:36.225333 1671 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:36.225435 1671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:36.225682 1671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:36.228056 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "7ddd4d086e3a49b7b278758740c18248"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "fe75cb47f11e700bcae710b7761586aa"
server_key_iv: "bca17b50957d0052e1724dfe922a874f"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.228669 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "7ddd4d086e3a49b7b278758740c18248"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "fe75cb47f11e700bcae710b7761586aa"
server_key_iv: "bca17b50957d0052e1724dfe922a874f"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.232761 1671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:36.235736 1686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.237185 1671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:36.237443 1671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "7ddd4d086e3a49b7b278758740c18248"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "fe75cb47f11e700bcae710b7761586aa"
server_key_iv: "bca17b50957d0052e1724dfe922a874f"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.237582 1671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:36.256188 1671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:36.259619 1671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:36.259871 1671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:36.260468 1671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:36.261262 1671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:36.261309 1671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.261349 1671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:36.261365 1671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.270610 1671 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:35643
I20260928 16:44:36.270628 1800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:35643 every 8 connection(s)
I20260928 16:44:36.271579 1671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:36.279848 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1671
I20260928 16:44:36.279978 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:36.280212 31504 external_mini_cluster.cc:1428] Setting key d45fe16ddb345a21e0cd3a9d5c3fac80
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:36.289202 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.273587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:34327 (local address 127.30.196.62:36663)
0928 16:44:36.273747 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.273751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.274733 (+ 982us) server_negotiation.cc:407] Connection header received
0928 16:44:36.276691 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.276695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.276749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.276847 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.278647 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.279363 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.280721 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.281017 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.281128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.284475 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:36.284478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:36.284480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:36.284525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:36.286312 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.287156 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.287159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.287161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.287267 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.287810 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.287813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.287814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.288020 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:36.288151 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:36.288927 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 16:44:36.289049 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:44:36.290555 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.274068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36663 (local address 127.30.196.2:34327)
0928 16:44:36.274546 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.274601 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.276440 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.277025 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.277034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.277761 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.278465 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.278483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.279526 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.279529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.280580 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.280591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.281195 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.281198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.281395 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.282165 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.282189 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.284250 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:36.286471 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.286482 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.286507 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.286975 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.287382 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.287388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.287393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.287642 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.288179 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:36.288189 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:36.288586 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 16:44:36.288854 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 16:44:36.289288 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:44:36.292131 1801 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36663
I20260928 16:44:36.292378 1801 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:36.292897 1801 heartbeater.cc:511] Master 127.30.196.62:36663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:36.294274 1473 ts_manager.cc:194] Registered new tserver with Master: 7ddd4d086e3a49b7b278758740c18248 (127.30.196.2:35643)
I20260928 16:44:36.295002 1473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:34327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:36.312539 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:36.431021 1806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:36.431330 1806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:36.431424 1806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:36.435997 1806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:36.436096 1806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:36.436190 1806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:36.441795 1806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:36.442961 1806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:36.444090 1806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:36.454423 1814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:36.454394 1812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:36.454596 1806 server_base.cc:1034] running on GCE node
W20260928 16:44:36.454394 1811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:36.455174 1806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:36.455914 1806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:36.457132 1806 hybrid_clock.cc:648] HybridClock initialized: now 1790613876457119 us; error 25 us; skew 500 ppm
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:36.460222 1806 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:36.461102 1806 webserver.cc:460] Webserver started at http://127.30.196.3:37113/ using document root <none> and password file <none>
I20260928 16:44:36.461671 1806 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:36.461761 1806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:36.461998 1806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:36.464107 1806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "bd4984691a9f46a5a50e4a5776c613ba"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "822c5d0c9938a82c036e6ba155496858"
server_key_iv: "bc9adb7ee33ebc84c3b12267da1d3024"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.464702 1806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "bd4984691a9f46a5a50e4a5776c613ba"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "822c5d0c9938a82c036e6ba155496858"
server_key_iv: "bc9adb7ee33ebc84c3b12267da1d3024"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.468677 1806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:36.471146 1821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.472644 1806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:36.472805 1806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "bd4984691a9f46a5a50e4a5776c613ba"
format_stamp: "Formatted at 2026-09-28 16:44:36 on dist-test-slave-5rg1"
server_key: "822c5d0c9938a82c036e6ba155496858"
server_key_iv: "bc9adb7ee33ebc84c3b12267da1d3024"
server_key_version: "encryptionkey@0"
I20260928 16:44:36.472923 1806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:36.486207 1806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:36.489449 1806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:36.489674 1806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:36.490293 1806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:36.491143 1806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:36.491190 1806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.491262 1806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:36.491300 1806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:36.500759 1806 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:45189
I20260928 16:44:36.500833 1935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:45189 every 8 connection(s)
I20260928 16:44:36.501701 1806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:36.509531 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1806
I20260928 16:44:36.509634 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:36.509958 31504 external_mini_cluster.cc:1428] Setting key a8067726b31282062944418b7f634272
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613876, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:36.517490 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.503608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:36449 (local address 127.30.196.62:36663)
0928 16:44:36.503775 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.503778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.504657 (+ 879us) server_negotiation.cc:407] Connection header received
0928 16:44:36.506137 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.506140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.506215 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.506310 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.507594 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.508221 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.509502 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.509793 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.509888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.513047 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:36.513050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:36.513053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:36.513100 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:36.514620 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.515323 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.515328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.515330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.515444 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:36.515917 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:36.515920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:36.515922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:36.516248 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:36.516371 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:36.517099 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 16:44:36.517250 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:44:36.518675 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.504047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36663 (local address 127.30.196.3:36449)
0928 16:44:36.504492 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.504534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.505897 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.506490 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.506497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.506858 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.507420 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.507434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.508356 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.508367 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.509345 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.509357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.509958 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.509963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.510173 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.510880 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.510909 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.512884 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:36.514752 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.514760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.514780 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.515190 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.515556 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:36.515561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:36.515564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:36.515804 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:36.516392 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:36.516400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:36.516768 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 16:44:36.517014 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 16:44:36.517448 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260928 16:44:36.520051 1936 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36663
I20260928 16:44:36.520335 1936 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:36.521763 1936 heartbeater.cc:511] Master 127.30.196.62:36663 requested a full tablet report, sending...
I20260928 16:44:36.523128 1473 ts_manager.cc:194] Registered new tserver with Master: bd4984691a9f46a5a50e4a5776c613ba (127.30.196.3:45189)
I20260928 16:44:36.523952 1473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:36449
I20260928 16:44:36.525995 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:44:36.541388 1513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:36.539732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.30.196.62:36663)
0928 16:44:36.539905 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.539910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.540071 (+ 161us) server_negotiation.cc:407] Connection header received
0928 16:44:36.540259 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.540265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.540317 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.540409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.541228 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39026: BlockingRecv error: recv got EOF from 127.0.0.1:39026 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:36.542680 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1536
I20260928 16:44:36.549381 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1671
I20260928 16:44:36.555205 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1806
I20260928 16:44:36.560863 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1442
2026-09-28T16:44:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2211 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1948](info): set up 2 sockets
Sep 28 16:44:36 dist-test-slave-5rg1 krb5kdc[1948](info): commencing operation
krb5kdc: starting...
W20260928 16:44:37.624912 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:37 dist-test-slave-5rg1 krb5kdc[1948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:37.760294 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:44897
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:37.877007 1961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:37.877313 1961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:37.877357 1961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:37.881647 1961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:37.881733 1961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:37.881753 1961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:37.881771 1961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:37.881788 1961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:37.887650 1961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:44897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:37.888919 1961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:37.890021 1961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:37.899521 1967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:37.899513 1966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:37.899572 1969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:37.899868 1961 server_base.cc:1034] running on GCE node
I20260928 16:44:37.900503 1961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:37.901708 1961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:37.902918 1961 hybrid_clock.cc:648] HybridClock initialized: now 1790613877902901 us; error 41 us; skew 500 ppm
Sep 28 16:44:37 dist-test-slave-5rg1 krb5kdc[1948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613877, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:37.905836 1961 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:37.906841 1961 webserver.cc:460] Webserver started at http://127.30.196.62:37687/ using document root <none> and password file <none>
I20260928 16:44:37.907436 1961 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:37.907531 1961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:37.907819 1961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:37.909991 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "9354a2175e164692842bcd9881690cd4"
format_stamp: "Formatted at 2026-09-28 16:44:37 on dist-test-slave-5rg1"
server_key: "6d0a930a077e918d21c8500072abeab9"
server_key_iv: "471b7efe64dfb8c348c4f89e2de8ae3e"
server_key_version: "encryptionkey@0"
I20260928 16:44:37.910596 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "9354a2175e164692842bcd9881690cd4"
format_stamp: "Formatted at 2026-09-28 16:44:37 on dist-test-slave-5rg1"
server_key: "6d0a930a077e918d21c8500072abeab9"
server_key_iv: "471b7efe64dfb8c348c4f89e2de8ae3e"
server_key_version: "encryptionkey@0"
I20260928 16:44:37.914574 1961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:37.917124 1976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:37.918426 1961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:44:37.918576 1961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "9354a2175e164692842bcd9881690cd4"
format_stamp: "Formatted at 2026-09-28 16:44:37 on dist-test-slave-5rg1"
server_key: "6d0a930a077e918d21c8500072abeab9"
server_key_iv: "471b7efe64dfb8c348c4f89e2de8ae3e"
server_key_version: "encryptionkey@0"
I20260928 16:44:37.918704 1961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:37.940752 1961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:37.944285 1961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:37.944519 1961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:37.952013 1961 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:44897
I20260928 16:44:37.952349 2028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:44897 every 8 connection(s)
I20260928 16:44:37.953075 1961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:37.956231 2029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:37.957819 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 1961
I20260928 16:44:37.957942 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:37.958241 31504 external_mini_cluster.cc:1428] Setting key 4720b9202d54bba70be27a2a5881c093
I20260928 16:44:37.962988 2029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4: Bootstrap starting.
I20260928 16:44:37.965451 2029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:37.966398 2029 log.cc:824] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:44:37 dist-test-slave-5rg1 krb5kdc[1948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:37.969014 2029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4: No bootstrap required, opened a new log
I20260928 16:44:37.973551 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:37.959735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.30.196.62:44897)
0928 16:44:37.960275 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:37.960294 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:37.960336 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:44:37.961020 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:37.961038 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:37.961401 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:37.961777 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:37.962685 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.963832 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.964741 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.965236 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.965414 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.968510 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:37.968516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:37.968537 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:37.968566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:37.970425 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:37.970954 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:37.970959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:37.970964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:37.971076 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:37.971452 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:37.971454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:37.971455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:37.971943 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:37.972132 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:37.972399 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 16:44:37.972714 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":159,"threads_started":1}
I20260928 16:44:37.973524 2029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } }
I20260928 16:44:37.973814 2029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:37.973884 2029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9354a2175e164692842bcd9881690cd4, State: Initialized, Role: FOLLOWER
I20260928 16:44:37.974272 2029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } }
I20260928 16:44:37.974382 2029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:37.974422 2029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:37.974486 2029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:37.975523 2029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } }
I20260928 16:44:37.975899 2029 leader_election.cc:302] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9354a2175e164692842bcd9881690cd4; no voters:
I20260928 16:44:37.976292 2029 leader_election.cc:288] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:37.976356 2034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:37.976540 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:37.976542 2034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 9354a2175e164692842bcd9881690cd4, State: Running, Role: LEADER
I20260928 16:44:37.976840 2034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } }
I20260928 16:44:37.977593 2029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:37.978649 2036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9354a2175e164692842bcd9881690cd4. Latest consensus state: current_term: 1 leader_uuid: "9354a2175e164692842bcd9881690cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } } }
I20260928 16:44:37.978809 2036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:37.979110 2035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9354a2175e164692842bcd9881690cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9354a2175e164692842bcd9881690cd4" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44897 } } }
I20260928 16:44:37.979223 2035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:37.979367 2044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:37.982992 2044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:37.989414 2044 catalog_manager.cc:1347] Generated new cluster ID: 547b2ff0214b4aa5975fa8e368842661
I20260928 16:44:37.989504 2044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:37.998652 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:37.992170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.30.196.62:44897)
0928 16:44:37.992316 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:37.992321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:37.992616 (+ 295us) server_negotiation.cc:407] Connection header received
0928 16:44:37.992784 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:37.992788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:37.992838 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:37.992933 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:37.993639 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.994454 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.995836 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.996215 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.996316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.997385 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:37.997390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:37.997392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:37.998004 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:37.998041 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:37.998060 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:37.998148 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:37.998298 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:44:37.998369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:38.041903 2044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:38.042975 2044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:38.049870 2044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9354a2175e164692842bcd9881690cd4: Generated new TSK 0
I20260928 16:44:38.050765 2044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:38.062606 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 1961
2026-09-28T16:44:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1500 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:38 dist-test-slave-5rg1 krb5kdc[2062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:38 dist-test-slave-5rg1 krb5kdc[2062](info): set up 2 sockets
Sep 28 16:44:38 dist-test-slave-5rg1 krb5kdc[2062](info): commencing operation
krb5kdc: starting...
W20260928 16:44:39.120118 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:39.255550 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38171
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:39.377967 2075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:39.378276 2075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:39.378332 2075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:39.382795 2075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:39.382869 2075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:39.382889 2075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:39.382907 2075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:39.382961 2075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:39.388578 2075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:39.389855 2075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:39.390947 2075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:39.401216 2075 server_base.cc:1034] running on GCE node
W20260928 16:44:39.401118 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.401237 2083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.401098 2080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:39.402208 2075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:39.403460 2075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:39.404692 2075 hybrid_clock.cc:648] HybridClock initialized: now 1790613879404677 us; error 38 us; skew 500 ppm
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:39.408190 2075 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:39.409294 2075 webserver.cc:460] Webserver started at http://127.30.196.62:41059/ using document root <none> and password file <none>
I20260928 16:44:39.409966 2075 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:39.410077 2075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:39.410342 2075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:39.412881 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "24360a59f3da4916a04de14ec5193f5a"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "05336f828c62b5a995c1418c4bf80054"
server_key_iv: "a5af5d8214c8b81b1f5fe70f95c30f10"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.413547 2075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "24360a59f3da4916a04de14ec5193f5a"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "05336f828c62b5a995c1418c4bf80054"
server_key_iv: "a5af5d8214c8b81b1f5fe70f95c30f10"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.417794 2075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:39.420720 2090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.422338 2075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:39.422492 2075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "24360a59f3da4916a04de14ec5193f5a"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "05336f828c62b5a995c1418c4bf80054"
server_key_iv: "a5af5d8214c8b81b1f5fe70f95c30f10"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.422629 2075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:39.444507 2075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:39.447952 2075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:39.448165 2075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:39.455552 2075 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38171
I20260928 16:44:39.456008 2142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38171 every 8 connection(s)
I20260928 16:44:39.456658 2075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:39.459547 2143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:39.463172 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2075
I20260928 16:44:39.463310 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:39.463610 31504 external_mini_cluster.cc:1428] Setting key 2f1945a8a6489f83bfeb6ba661d22a7e
I20260928 16:44:39.465909 2143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a: Bootstrap starting.
I20260928 16:44:39.468524 2143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:39.469385 2143 log.cc:824] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:39.471477 2143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a: No bootstrap required, opened a new log
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:39.474958 2143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } }
I20260928 16:44:39.475246 2143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:39.475337 2143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24360a59f3da4916a04de14ec5193f5a, State: Initialized, Role: FOLLOWER
I20260928 16:44:39.475833 2143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } }
I20260928 16:44:39.475958 2143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:39.476056 2143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:39.476190 2143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:39.477120 2143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } }
I20260928 16:44:39.477463 2143 leader_election.cc:302] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24360a59f3da4916a04de14ec5193f5a; no voters:
I20260928 16:44:39.477857 2143 leader_election.cc:288] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:39.477973 2148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:39.478240 2148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [term 1 LEADER]: Becoming Leader. State: Replica: 24360a59f3da4916a04de14ec5193f5a, State: Running, Role: LEADER
I20260928 16:44:39.478601 2148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } }
I20260928 16:44:39.479179 2143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:39.479496 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.465177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.30.196.62:38171)
0928 16:44:39.465617 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.465633 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.465669 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:44:39.466369 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.466388 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.466721 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.466993 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.467939 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.469032 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.470148 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.470620 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.470800 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.473615 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:39.473620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:39.473634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:39.473662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:39.476166 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.476652 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.476658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.476664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.476784 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.477076 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.477080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.477081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.477546 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:39.477750 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:39.478220 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 16:44:39.478547 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260928 16:44:39.479838 2148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24360a59f3da4916a04de14ec5193f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } } }
I20260928 16:44:39.479956 2148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:39.480224 2148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24360a59f3da4916a04de14ec5193f5a. Latest consensus state: current_term: 1 leader_uuid: "24360a59f3da4916a04de14ec5193f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24360a59f3da4916a04de14ec5193f5a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38171 } } }
I20260928 16:44:39.480357 2148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:39.480643 2155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:39.483817 2155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:39.489682 2155 catalog_manager.cc:1347] Generated new cluster ID: 689915502b9a4082b142fdbdab707b72
I20260928 16:44:39.489766 2155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:39.507709 2155 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:39.509073 2155 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:39.517297 2155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24360a59f3da4916a04de14ec5193f5a: Generated new TSK 0
I20260928 16:44:39.518205 2155 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:39.552323 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:39.667631 2169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:39.667924 2169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:39.667968 2169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:39.672171 2169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:39.672245 2169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:39.672307 2169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:39.677759 2169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:39.678875 2169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:39.680024 2169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:39.690227 2177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.690218 2174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.690217 2175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:39.690475 2169 server_base.cc:1034] running on GCE node
I20260928 16:44:39.691000 2169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:39.691789 2169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:39.693058 2169 hybrid_clock.cc:648] HybridClock initialized: now 1790613879693042 us; error 27 us; skew 500 ppm
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:39.696026 2169 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:39.696923 2169 webserver.cc:460] Webserver started at http://127.30.196.1:37367/ using document root <none> and password file <none>
I20260928 16:44:39.697458 2169 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:39.697521 2169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:39.697690 2169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:39.699810 2169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "24ee9c2f0f1f46c39debe1273837a975"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "009e21dadfb4e7cf49916b6c38f0e371"
server_key_iv: "7f29111d9ac407bfeacf4488da3fbf6b"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.700409 2169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "24ee9c2f0f1f46c39debe1273837a975"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "009e21dadfb4e7cf49916b6c38f0e371"
server_key_iv: "7f29111d9ac407bfeacf4488da3fbf6b"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.704362 2169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:39.706811 2184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.708268 2169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:39.708419 2169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "24ee9c2f0f1f46c39debe1273837a975"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "009e21dadfb4e7cf49916b6c38f0e371"
server_key_iv: "7f29111d9ac407bfeacf4488da3fbf6b"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.708541 2169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:39.730716 2169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:39.734359 2169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:39.734577 2169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:39.735209 2169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:39.736222 2169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:39.736297 2169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.736370 2169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:39.736423 2169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.746069 2169 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:36963
I20260928 16:44:39.746093 2297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:36963 every 8 connection(s)
I20260928 16:44:39.747021 2169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:39.749498 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2169
I20260928 16:44:39.749650 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:39.749954 31504 external_mini_cluster.cc:1428] Setting key 2ab40bf0f59ecde563bb414612dac95b
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:39.763115 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.749067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:36635 (local address 127.30.196.62:38171)
0928 16:44:39.749223 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.749227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.750218 (+ 991us) server_negotiation.cc:407] Connection header received
0928 16:44:39.751722 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.751726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.751781 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.751874 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.753123 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.753741 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.755515 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.755893 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.756008 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.759169 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:39.759172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:39.759175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:39.759199 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:39.760827 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.761445 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.761449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.761451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.761538 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.761916 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.761919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.761921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.762139 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:39.762250 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:39.762797 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 16:44:39.762940 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:39.764259 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.749523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38171 (local address 127.30.196.1:36635)
0928 16:44:39.750031 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:39.750087 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:39.751429 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:39.752047 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:39.752054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:39.752431 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:39.752955 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.752968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.753887 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.753896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.755358 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.755374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.756095 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.756100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.756338 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.757035 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:39.757064 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:39.758970 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:39.761004 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:39.761011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:39.761029 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:39.761325 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:39.761651 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:39.761654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:39.761656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:39.761812 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:39.762266 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:39.762274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:39.762546 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 16:44:39.762756 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 16:44:39.763139 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260928 16:44:39.765715 2298 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38171
I20260928 16:44:39.765986 2298 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:39.766510 2298 heartbeater.cc:511] Master 127.30.196.62:38171 requested a full tablet report, sending...
I20260928 16:44:39.768391 2107 ts_manager.cc:194] Registered new tserver with Master: 24ee9c2f0f1f46c39debe1273837a975 (127.30.196.1:36963)
I20260928 16:44:39.770201 2107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:36635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:39.783262 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:39.896997 2303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:39.897290 2303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:39.897336 2303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:39.901391 2303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:39.901459 2303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:39.901522 2303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:39.907105 2303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:39.908246 2303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:39.909350 2303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:39.919983 2303 server_base.cc:1034] running on GCE node
W20260928 16:44:39.920084 2308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.919934 2311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:39.920446 2309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:39.920877 2303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:39.921556 2303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:39.922719 2303 hybrid_clock.cc:648] HybridClock initialized: now 1790613879922692 us; error 38 us; skew 500 ppm
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:39.925589 2303 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:39.926559 2303 webserver.cc:460] Webserver started at http://127.30.196.2:46037/ using document root <none> and password file <none>
I20260928 16:44:39.927201 2303 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:39.927269 2303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:39.927438 2303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:39.929533 2303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "098dec76624a4555848c6e3f1b99dff0"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "6740251fd35f3336a8e8b81e4c35a875"
server_key_iv: "d7b487690eb7035ce33c3e3ade848588"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.930061 2303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "098dec76624a4555848c6e3f1b99dff0"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "6740251fd35f3336a8e8b81e4c35a875"
server_key_iv: "d7b487690eb7035ce33c3e3ade848588"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.933799 2303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:39.936296 2318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.937616 2303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:39.937721 2303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "098dec76624a4555848c6e3f1b99dff0"
format_stamp: "Formatted at 2026-09-28 16:44:39 on dist-test-slave-5rg1"
server_key: "6740251fd35f3336a8e8b81e4c35a875"
server_key_iv: "d7b487690eb7035ce33c3e3ade848588"
server_key_version: "encryptionkey@0"
I20260928 16:44:39.937865 2303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:39.963496 2303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:39.966886 2303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:39.967105 2303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:39.967762 2303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:39.968641 2303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:39.968691 2303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.968731 2303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:39.968787 2303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:39.978330 2303 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:37141
I20260928 16:44:39.978430 2432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:37141 every 8 connection(s)
I20260928 16:44:39.979385 2303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:39.980743 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2303
I20260928 16:44:39.980862 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:39.981189 31504 external_mini_cluster.cc:1428] Setting key 4d6a0f35f975191c82c29234661f825f
Sep 28 16:44:39 dist-test-slave-5rg1 krb5kdc[2062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613879, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:39.995985 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.981627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:35109 (local address 127.30.196.62:38171)
0928 16:44:39.981771 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.981781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.982885 (+ 1104us) server_negotiation.cc:407] Connection header received
0928 16:44:39.984567 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.984569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.984618 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.984703 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.986389 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.987018 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.988486 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.988928 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.989066 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.991782 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:39.991785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:39.991788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:39.991813 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:39.993193 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.993875 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.993878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.993879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.993967 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:39.994453 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:39.994460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:39.994462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:39.994753 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:39.994860 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:39.995623 (+ 763us) server_negotiation.cc:299] Negotiation successful
0928 16:44:39.995805 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:44:39.996986 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.982098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38171 (local address 127.30.196.2:35109)
0928 16:44:39.982709 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:39.982763 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:39.984294 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:39.984930 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:39.984949 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:39.985495 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:39.986204 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.986234 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.987178 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.987184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.988351 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.988361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.989093 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.989095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.989244 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.989898 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:39.989916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:39.991566 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:39.993471 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:39.993484 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:39.993498 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:39.993766 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:39.994087 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:39.994090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:39.994091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:39.994321 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:39.994958 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:39.994966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:39.995324 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 16:44:39.995549 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 16:44:39.995982 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":126,"threads_started":1}
I20260928 16:44:39.998411 2433 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38171
I20260928 16:44:39.998684 2433 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:39.999339 2433 heartbeater.cc:511] Master 127.30.196.62:38171 requested a full tablet report, sending...
I20260928 16:44:40.000670 2107 ts_manager.cc:194] Registered new tserver with Master: 098dec76624a4555848c6e3f1b99dff0 (127.30.196.2:37141)
I20260928 16:44:40.001291 2107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:35109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:40.016553 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:40.135269 2438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:40.135596 2438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:40.135671 2438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:40.139799 2438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:40.139882 2438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:40.139977 2438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:40.145663 2438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:40.146799 2438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:40.147920 2438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:40.157857 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:40.157845 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:40.157872 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:40.158084 2438 server_base.cc:1034] running on GCE node
I20260928 16:44:40.158623 2438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:40.159318 2438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:40.160518 2438 hybrid_clock.cc:648] HybridClock initialized: now 1790613880160494 us; error 34 us; skew 500 ppm
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:40.163355 2438 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:40.164290 2438 webserver.cc:460] Webserver started at http://127.30.196.3:40361/ using document root <none> and password file <none>
I20260928 16:44:40.164856 2438 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:40.164948 2438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:40.165169 2438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:40.167191 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "4f8936eeaef1462987ee5081e8f78b9c"
format_stamp: "Formatted at 2026-09-28 16:44:40 on dist-test-slave-5rg1"
server_key: "0553717ef99b7dd41314dbf330d87b52"
server_key_iv: "9a1b1d063ec152c49b7bdabbbd839335"
server_key_version: "encryptionkey@0"
I20260928 16:44:40.167807 2438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8936eeaef1462987ee5081e8f78b9c"
format_stamp: "Formatted at 2026-09-28 16:44:40 on dist-test-slave-5rg1"
server_key: "0553717ef99b7dd41314dbf330d87b52"
server_key_iv: "9a1b1d063ec152c49b7bdabbbd839335"
server_key_version: "encryptionkey@0"
I20260928 16:44:40.171591 2438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:40.174167 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:40.175457 2438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:40.175626 2438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "4f8936eeaef1462987ee5081e8f78b9c"
format_stamp: "Formatted at 2026-09-28 16:44:40 on dist-test-slave-5rg1"
server_key: "0553717ef99b7dd41314dbf330d87b52"
server_key_iv: "9a1b1d063ec152c49b7bdabbbd839335"
server_key_version: "encryptionkey@0"
I20260928 16:44:40.175810 2438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:40.196998 2438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:40.201171 2438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:40.201411 2438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:40.202080 2438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:40.203027 2438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:40.203119 2438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:40.203225 2438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:40.203333 2438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 16:44:40.213280 2438 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:37117
I20260928 16:44:40.213339 2566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:37117 every 8 connection(s)
I20260928 16:44:40.214324 2438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:40.224316 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2438
I20260928 16:44:40.224479 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:40.224725 31504 external_mini_cluster.cc:1428] Setting key 2f795b54d3b157fe393ef1d91af25178
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613880, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:40.229424 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.216354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:42209 (local address 127.30.196.62:38171)
0928 16:44:40.216502 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.216506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.217371 (+ 865us) server_negotiation.cc:407] Connection header received
0928 16:44:40.218896 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.218900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.218974 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.219083 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.220403 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.220955 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.222116 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.222336 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.222456 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.225519 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:40.225523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:40.225526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:40.225560 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:40.227160 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:40.227799 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:40.227803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:40.227813 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:40.227915 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:40.228280 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:40.228283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:40.228285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:40.228474 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:40.228583 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:40.229117 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 16:44:40.229245 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:40.230345 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.216737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38171 (local address 127.30.196.3:42209)
0928 16:44:40.217208 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:40.217252 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:40.218650 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:40.219267 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:40.219273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:40.219726 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:40.220244 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:40.220258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.221090 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.221099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:40.221984 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:40.221994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.222504 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.222508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:40.222684 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.223510 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:40.223540 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:40.225332 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:40.227280 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:40.227287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:40.227303 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:40.227710 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:40.228019 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:40.228023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:40.228025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:40.228176 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:40.228583 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:40.228591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:40.228856 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 16:44:40.229061 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 16:44:40.229423 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260928 16:44:40.231501 2567 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38171
I20260928 16:44:40.231804 2567 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:40.232311 2567 heartbeater.cc:511] Master 127.30.196.62:38171 requested a full tablet report, sending...
I20260928 16:44:40.233358 2107 ts_manager.cc:194] Registered new tserver with Master: 4f8936eeaef1462987ee5081e8f78b9c (127.30.196.3:37117)
I20260928 16:44:40.233897 2107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:42209
I20260928 16:44:40.238716 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613880, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613880, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:40.267073 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.257159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.30.196.62:38171)
0928 16:44:40.257369 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.257373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.257414 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:44:40.257553 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.257556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.257604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.257683 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.258410 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.259021 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.260113 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.260330 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.260424 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.263335 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:40.263339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:40.263341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:40.263372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:40.265079 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:40.265605 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:40.265617 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:40.265619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:40.265702 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:40.266026 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:40.266031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:40.266033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:40.266299 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:40.266463 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:40.266744 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 16:44:40.266882 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 16:44:40.267616 2107 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48352
I20260928 16:44:40.268698 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2169
I20260928 16:44:40.274863 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2303
I20260928 16:44:40.281476 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2438
I20260928 16:44:40.287568 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2075
2026-09-28T16:44:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2228 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2579](info): set up 2 sockets
Sep 28 16:44:40 dist-test-slave-5rg1 krb5kdc[2579](info): commencing operation
krb5kdc: starting...
W20260928 16:44:41.352268 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:41 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:41.487030 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:41613
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:41.601195 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:41.601496 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:41.601537 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:41.605747 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:41.605825 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:41.605844 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:41.605861 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:41.605876 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:41.611759 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:41.612988 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:41.614089 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:41.623293 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.623381 2592 server_base.cc:1034] running on GCE node
W20260928 16:44:41.623293 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.623293 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.624306 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:41.625556 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:41.626791 2592 hybrid_clock.cc:648] HybridClock initialized: now 1790613881626771 us; error 34 us; skew 500 ppm
Sep 28 16:44:41 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613881, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:41.629979 2592 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:41.631121 2592 webserver.cc:460] Webserver started at http://127.30.196.62:34165/ using document root <none> and password file <none>
I20260928 16:44:41.631811 2592 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:41.631914 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:41.632174 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:41.634400 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "49fa4a52c2184709a38606b1cf129956"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "fd8e16b6f5671aa36d9c1c6ad1b128d4"
server_key_iv: "c5d2c4a08d28ed4ea0c3dba3d3d6a0d3"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.635003 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "49fa4a52c2184709a38606b1cf129956"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "fd8e16b6f5671aa36d9c1c6ad1b128d4"
server_key_iv: "c5d2c4a08d28ed4ea0c3dba3d3d6a0d3"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.638957 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:41.641570 2607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.642830 2592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:41.642987 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "49fa4a52c2184709a38606b1cf129956"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "fd8e16b6f5671aa36d9c1c6ad1b128d4"
server_key_iv: "c5d2c4a08d28ed4ea0c3dba3d3d6a0d3"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.643126 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:41.662768 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:41.666052 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:41.666287 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:41.674387 2592 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:41613
I20260928 16:44:41.674410 2659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:41613 every 8 connection(s)
I20260928 16:44:41.675395 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:41.678352 2660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:41.684276 2660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: Bootstrap starting.
I20260928 16:44:41.684386 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2592
I20260928 16:44:41.684509 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:41.684798 31504 external_mini_cluster.cc:1428] Setting key d7a43c9cdf4d308947b63640fb9b02fe
I20260928 16:44:41.687095 2660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:41.687942 2660 log.cc:824] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:41.689818 2660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: No bootstrap required, opened a new log
I20260928 16:44:41.693645 2660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } }
I20260928 16:44:41.693956 2660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:41.694010 2660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fa4a52c2184709a38606b1cf129956, State: Initialized, Role: FOLLOWER
Sep 28 16:44:41 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:41.694521 2660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } }
I20260928 16:44:41.694684 2660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:41.694770 2660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:41.694900 2660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:41.696197 2660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } }
I20260928 16:44:41.696580 2660 leader_election.cc:302] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49fa4a52c2184709a38606b1cf129956; no voters:
I20260928 16:44:41.696981 2660 leader_election.cc:288] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:41.697096 2665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:41.697338 2665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [term 1 LEADER]: Becoming Leader. State: Replica: 49fa4a52c2184709a38606b1cf129956, State: Running, Role: LEADER
I20260928 16:44:41.697703 2665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } }
I20260928 16:44:41.698333 2660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:41.699419 2666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49fa4a52c2184709a38606b1cf129956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } } }
I20260928 16:44:41.699591 2666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:41.699743 2667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49fa4a52c2184709a38606b1cf129956. Latest consensus state: current_term: 1 leader_uuid: "49fa4a52c2184709a38606b1cf129956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fa4a52c2184709a38606b1cf129956" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 41613 } } }
I20260928 16:44:41.699828 2667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:41.700327 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:41.686302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.30.196.62:41613)
0928 16:44:41.686796 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:41.686812 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:41.686847 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:44:41.687574 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:41.687601 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:41.688024 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:41.688398 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:41.689231 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.690273 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.691162 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.691647 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.691888 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:41.694699 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:41.694706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:41.694725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:41.694760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:41.696680 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:41.697297 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:41.697303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:41.697309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:41.697450 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:41.697754 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:41.697758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:41.697760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:41.698275 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:41.698468 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:41.698820 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:44:41.699148 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":149,"threads_started":1}
W20260928 16:44:41.703399 2680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:41.703470 2680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:41.703553 2674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:41.704385 2674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:41.710441 2674 catalog_manager.cc:1347] Generated new cluster ID: 68d5cc417a69422baf395bb5b0464b92
I20260928 16:44:41.710546 2674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:41.719306 2674 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:41.720245 2674 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:41.728917 2674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49fa4a52c2184709a38606b1cf129956: Generated new TSK 0
I20260928 16:44:41.729709 2674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:41.766696 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:41.907478 2686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:41.907878 2686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:41.907985 2686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:41.912433 2686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:41.912544 2686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:41.912636 2686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:41.918468 2686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:41.919644 2686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:41.920776 2686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:41.931216 2694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.931337 2691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.931401 2686 server_base.cc:1034] running on GCE node
W20260928 16:44:41.931337 2692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.932034 2686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:41.932751 2686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:41.933938 2686 hybrid_clock.cc:648] HybridClock initialized: now 1790613881933901 us; error 46 us; skew 500 ppm
Sep 28 16:44:41 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613881, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:41.936836 2686 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:41.937801 2686 webserver.cc:460] Webserver started at http://127.30.196.1:40555/ using document root <none> and password file <none>
I20260928 16:44:41.938392 2686 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:41.938462 2686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:41.938730 2686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:41.940989 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "ad3f63d9cb9c4db3a2e370be9d010c91"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "e7eddead70ec7d3e7859b41190b1a0a0"
server_key_iv: "5b0746633b8c273e70f6b8c88a35e638"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.941612 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "ad3f63d9cb9c4db3a2e370be9d010c91"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "e7eddead70ec7d3e7859b41190b1a0a0"
server_key_iv: "5b0746633b8c273e70f6b8c88a35e638"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.945658 2686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:44:41.948453 2701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.949867 2686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:41.950022 2686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "ad3f63d9cb9c4db3a2e370be9d010c91"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-5rg1"
server_key: "e7eddead70ec7d3e7859b41190b1a0a0"
server_key_iv: "5b0746633b8c273e70f6b8c88a35e638"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.950147 2686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:41.963284 2686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:41.966337 2686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:41.966567 2686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:41.967201 2686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:41.968123 2686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:41.968171 2686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.968240 2686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:41.968271 2686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.977486 2686 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45927
I20260928 16:44:41.977509 2814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45927 every 8 connection(s)
I20260928 16:44:41.978421 2686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:41.984722 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2686
I20260928 16:44:41.984853 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:41.985260 31504 external_mini_cluster.cc:1428] Setting key cdc7f4875ac6571452739e3bba9b8a8a
Sep 28 16:44:41 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613881, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:41.993850 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:41.980508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:53781 (local address 127.30.196.62:41613)
0928 16:44:41.980648 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:41.980652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:41.981570 (+ 918us) server_negotiation.cc:407] Connection header received
0928 16:44:41.983117 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:41.983121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:41.983171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:41.983253 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:41.984788 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.985601 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.986972 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.987356 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.987513 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:41.990236 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:41.990239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:41.990241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:41.990265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:41.991770 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:41.992295 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:41.992300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:41.992302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:41.992382 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:41.992735 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:41.992738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:41.992740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:41.992919 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:41.993030 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:41.993543 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 16:44:41.993671 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:41.994729 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:41.980873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:41613 (local address 127.30.196.1:53781)
0928 16:44:41.981412 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:41.981456 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:41.982844 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:41.983468 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:41.983477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:41.984039 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:41.984583 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:41.984606 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.985774 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.985780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:41.986837 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:41.986848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.987516 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.987518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:41.987770 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:41.988413 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:41.988430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:41.990054 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:41.991888 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:41.991894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:41.991909 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:41.992189 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:41.992478 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:41.992484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:41.992486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:41.992629 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:41.993031 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:41.993038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:41.993294 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 16:44:41.993481 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:44:41.993812 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":148,"threads_started":1}
I20260928 16:44:41.996068 2815 heartbeater.cc:347] Connected to a master server at 127.30.196.62:41613
I20260928 16:44:41.996299 2815 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:41.996887 2815 heartbeater.cc:511] Master 127.30.196.62:41613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:41.998696 2624 ts_manager.cc:194] Registered new tserver with Master: ad3f63d9cb9c4db3a2e370be9d010c91 (127.30.196.1:45927)
I20260928 16:44:42.000072 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:53781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:42.014858 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:42.131306 2820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:42.131616 2820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:42.131718 2820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:42.136066 2820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:42.136137 2820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:42.136237 2820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:42.141955 2820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:42.143340 2820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:42.144618 2820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:42.155350 2828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.155435 2820 server_base.cc:1034] running on GCE node
W20260928 16:44:42.155352 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.155352 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.156154 2820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:42.156880 2820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:42.158087 2820 hybrid_clock.cc:648] HybridClock initialized: now 1790613882158068 us; error 34 us; skew 500 ppm
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:42.161029 2820 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:42.162029 2820 webserver.cc:460] Webserver started at http://127.30.196.2:42041/ using document root <none> and password file <none>
I20260928 16:44:42.162642 2820 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:42.162737 2820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:42.162976 2820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:42.165127 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "5e5390fac92b4fd196924b0982ab2e69"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "ca0e880290d27466c1d0389aa4bbc65b"
server_key_iv: "5a1ad4710bf71aefeaf72e2bfde772c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.165735 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "5e5390fac92b4fd196924b0982ab2e69"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "ca0e880290d27466c1d0389aa4bbc65b"
server_key_iv: "5a1ad4710bf71aefeaf72e2bfde772c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.169564 2820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:42.172099 2835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.173324 2820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:42.173470 2820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "5e5390fac92b4fd196924b0982ab2e69"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "ca0e880290d27466c1d0389aa4bbc65b"
server_key_iv: "5a1ad4710bf71aefeaf72e2bfde772c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.173597 2820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:42.214869 2820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:42.218228 2820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:42.218469 2820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:42.219096 2820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:42.220048 2820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:42.220119 2820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.220196 2820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:42.220248 2820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.229503 2820 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38223
I20260928 16:44:42.229517 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38223 every 8 connection(s)
I20260928 16:44:42.230479 2820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:42.232597 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2820
I20260928 16:44:42.232748 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:42.233037 31504 external_mini_cluster.cc:1428] Setting key e024a228baf85e4cebfa12b08e91ec71
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:42.246536 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.232623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:46281 (local address 127.30.196.62:41613)
0928 16:44:42.232807 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.232812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.233592 (+ 780us) server_negotiation.cc:407] Connection header received
0928 16:44:42.235116 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.235120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.235174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.235253 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.236888 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.237422 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.239278 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.239491 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.239573 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.242658 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.242660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.242663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:42.242689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.244283 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.244849 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.244853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.244855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.244940 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.245304 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.245308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.245310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.245505 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.245623 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.246181 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.246346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:42.247445 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.232944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:41613 (local address 127.30.196.2:46281)
0928 16:44:42.233434 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:42.233490 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:42.234890 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:42.235424 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:42.235431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:42.235954 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:42.236710 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.236733 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.237567 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.237574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.239151 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.239163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.239694 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.239702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.239964 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.240674 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:42.240702 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:42.242451 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:42.244415 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:42.244422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:42.244439 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:42.244734 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:42.245040 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:42.245044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:42.245045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:42.245201 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:42.245619 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:42.245626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:42.245915 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 16:44:42.246140 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 16:44:42.246514 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":148,"threads_started":1}
I20260928 16:44:42.248821 2949 heartbeater.cc:347] Connected to a master server at 127.30.196.62:41613
I20260928 16:44:42.249169 2949 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:42.249771 2949 heartbeater.cc:511] Master 127.30.196.62:41613 requested a full tablet report, sending...
I20260928 16:44:42.250988 2624 ts_manager.cc:194] Registered new tserver with Master: 5e5390fac92b4fd196924b0982ab2e69 (127.30.196.2:38223)
I20260928 16:44:42.251621 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:46281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:42.266919 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:42.386998 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:42.387295 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:42.387341 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:42.391630 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:42.391737 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:42.391800 2954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:42.397354 2954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:42.398496 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:42.399590 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:42.409823 2959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.409775 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.409855 2954 server_base.cc:1034] running on GCE node
W20260928 16:44:42.409795 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.410423 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:42.411144 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:42.412371 2954 hybrid_clock.cc:648] HybridClock initialized: now 1790613882412347 us; error 41 us; skew 500 ppm
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:42.415334 2954 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:42.416344 2954 webserver.cc:460] Webserver started at http://127.30.196.3:37349/ using document root <none> and password file <none>
I20260928 16:44:42.416929 2954 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:42.416989 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:42.417239 2954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:42.419433 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "f88fe91ff3974ec9b02c1e5b3cec3054"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "80e84f767c6014d6a83c543114506100"
server_key_iv: "09fa0e79d635d7eeaa2f3f5723ff5b24"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.420058 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "f88fe91ff3974ec9b02c1e5b3cec3054"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "80e84f767c6014d6a83c543114506100"
server_key_iv: "09fa0e79d635d7eeaa2f3f5723ff5b24"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.423987 2954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:42.426578 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.428058 2954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:42.428190 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "f88fe91ff3974ec9b02c1e5b3cec3054"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-5rg1"
server_key: "80e84f767c6014d6a83c543114506100"
server_key_iv: "09fa0e79d635d7eeaa2f3f5723ff5b24"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.428308 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:42.443902 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:42.447532 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:42.447800 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:42.448477 2954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:42.449337 2954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:42.449383 2954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.449456 2954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:42.449494 2954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.459172 2954 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:35489
I20260928 16:44:42.459189 3082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:35489 every 8 connection(s)
I20260928 16:44:42.460232 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:42.464649 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 2954
I20260928 16:44:42.464799 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:42.465083 31504 external_mini_cluster.cc:1428] Setting key aac2655c564a3efc82167e1b3e7a4b2a
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:42.474807 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.462183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:50371 (local address 127.30.196.62:41613)
0928 16:44:42.462325 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.462329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.463091 (+ 762us) server_negotiation.cc:407] Connection header received
0928 16:44:42.464626 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.464629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.464695 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.464779 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.466228 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.466831 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.468039 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.468265 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.468426 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.471094 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.471097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.471099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:42.471125 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.472724 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.473264 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.473268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.473270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.473357 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.473703 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.473708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.473710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.473898 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.474009 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.474518 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.474643 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:42.475634 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.462465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:41613 (local address 127.30.196.3:50371)
0928 16:44:42.462926 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:42.462969 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:42.464377 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:42.464986 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:42.464993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:42.465453 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:42.466049 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.466070 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.466977 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.466980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.467901 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.467910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.468479 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.468483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.468652 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.469227 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:42.469244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:42.470838 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:42.472849 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:42.472855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:42.472873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:42.473154 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:42.473455 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:42.473459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:42.473461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:42.473605 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:42.474012 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:42.474021 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:42.474271 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 16:44:42.474469 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 16:44:42.474812 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260928 16:44:42.476900 3083 heartbeater.cc:347] Connected to a master server at 127.30.196.62:41613
I20260928 16:44:42.477205 3083 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:42.477734 3083 heartbeater.cc:511] Master 127.30.196.62:41613 requested a full tablet report, sending...
I20260928 16:44:42.478995 2624 ts_manager.cc:194] Registered new tserver with Master: f88fe91ff3974ec9b02c1e5b3cec3054 (127.30.196.3:35489)
I20260928 16:44:42.479645 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:42.479784 2624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:50371
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:42.509207 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.498496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.30.196.1:45927)
0928 16:44:42.498787 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.498795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.498825 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:44:42.498921 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.498926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.499130 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.499326 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.500234 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.501053 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.501946 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.502354 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.502485 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.505150 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.505162 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.505177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:42.505206 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.506998 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.507488 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.507494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.507495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.507587 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.507965 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.507968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.507970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.508261 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.508458 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.508759 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.508966 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[2579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:42.523161 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.512538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.30.196.62:41613)
0928 16:44:42.512680 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.512686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.512766 (+ 80us) server_negotiation.cc:407] Connection header received
0928 16:44:42.512945 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.512947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.512992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.513135 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.513781 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.514315 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.515733 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.515964 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.516091 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.519117 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.519123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.519129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:42.519170 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.521153 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.521794 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.521799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.521801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.521904 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:42.522255 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:42.522259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:42.522262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:42.522561 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.522686 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.522908 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.523016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:44:42.523646 2624 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52166
I20260928 16:44:42.524451 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2686
I20260928 16:44:42.532248 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2820
I20260928 16:44:42.538566 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2954
I20260928 16:44:42.544709 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 2592
2026-09-28T16:44:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2255 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[3095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[3095](info): set up 2 sockets
Sep 28 16:44:42 dist-test-slave-5rg1 krb5kdc[3095](info): commencing operation
krb5kdc: starting...
W20260928 16:44:43.600301 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:43 dist-test-slave-5rg1 krb5kdc[3095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:43.731905 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38515
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:43.845857 3109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:43.846158 3109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:43.846202 3109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:43.850406 3109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:43.850479 3109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:43.850499 3109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:43.850517 3109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:43.850533 3109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:43.855907 3109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:43.856986 3109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:43.858067 3109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:43.866729 3117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:43.866927 3109 server_base.cc:1034] running on GCE node
W20260928 16:44:43.866732 3114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:43.866750 3115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:43.867807 3109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:43.868993 3109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:43.870235 3109 hybrid_clock.cc:648] HybridClock initialized: now 1790613883870207 us; error 34 us; skew 500 ppm
Sep 28 16:44:43 dist-test-slave-5rg1 krb5kdc[3095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613883, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:43.873425 3109 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:43.874193 3109 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:43.874264 3109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:43.874437 3109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:43.876744 3109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "ae36c0776af44309873c06ea08b42f12"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-5rg1"
server_key: "5b74b6d8841ac114c1edbfa0c7156ed2"
server_key_iv: "41ad0c38f07074194bac98b10acd52c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.877275 3109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "ae36c0776af44309873c06ea08b42f12"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-5rg1"
server_key: "5b74b6d8841ac114c1edbfa0c7156ed2"
server_key_iv: "41ad0c38f07074194bac98b10acd52c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.881176 3109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:43.883791 3123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:43.885149 3109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:43.885301 3109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "ae36c0776af44309873c06ea08b42f12"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-5rg1"
server_key: "5b74b6d8841ac114c1edbfa0c7156ed2"
server_key_iv: "41ad0c38f07074194bac98b10acd52c7"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.885421 3109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:43.910254 3109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:43.913789 3109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:43.914021 3109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:43.921895 3109 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38515
I20260928 16:44:43.922228 3175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38515 every 8 connection(s)
I20260928 16:44:43.922616 3109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:43.925470 3176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:43.929204 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3109
I20260928 16:44:43.929373 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:43.929641 31504 external_mini_cluster.cc:1428] Setting key 715e9cf2ae30eb3eebc7958aed3f44f8
I20260928 16:44:43.931726 3176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: Bootstrap starting.
I20260928 16:44:43.934386 3176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:43.935197 3176 log.cc:824] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:43.937476 3176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: No bootstrap required, opened a new log
Sep 28 16:44:43 dist-test-slave-5rg1 krb5kdc[3095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:43.941779 3176 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } }
I20260928 16:44:43.942085 3176 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:43.942142 3176 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36c0776af44309873c06ea08b42f12, State: Initialized, Role: FOLLOWER
I20260928 16:44:43.942651 3176 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } }
I20260928 16:44:43.942790 3176 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:43.942885 3176 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:43.943002 3176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:43.944284 3176 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } }
I20260928 16:44:43.944669 3176 leader_election.cc:302] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae36c0776af44309873c06ea08b42f12; no voters:
I20260928 16:44:43.945017 3176 leader_election.cc:288] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:43.945156 3181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:43.945423 3181 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [term 1 LEADER]: Becoming Leader. State: Replica: ae36c0776af44309873c06ea08b42f12, State: Running, Role: LEADER
I20260928 16:44:43.945716 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:43.931075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.30.196.62:38515)
0928 16:44:43.931550 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:43.931566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:43.931610 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:44:43.932376 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:43.932394 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:43.932746 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:43.933035 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:43.933972 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.935087 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.936202 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.936698 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.936822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:43.939570 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:43.939576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:43.939594 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:43.939626 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:43.942348 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:43.942874 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:43.942880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:43.942887 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:43.943033 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:43.943358 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:43.943361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:43.943363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:43.943926 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:43.944143 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:43.944348 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 16:44:43.944675 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
I20260928 16:44:43.945823 3181 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } }
I20260928 16:44:43.946372 3176 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:43.947098 3182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae36c0776af44309873c06ea08b42f12. Latest consensus state: current_term: 1 leader_uuid: "ae36c0776af44309873c06ea08b42f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } } }
I20260928 16:44:43.947358 3182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:43.947517 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae36c0776af44309873c06ea08b42f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36c0776af44309873c06ea08b42f12" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38515 } } }
I20260928 16:44:43.947623 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:43.948057 3190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:43.951189 3190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:43.957181 3190 catalog_manager.cc:1347] Generated new cluster ID: 04da2a76498f4d3dbdda6da2dcf64a9e
I20260928 16:44:43.957270 3190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:43.970484 3190 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:43.971717 3190 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:43.977734 3190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: Generated new TSK 0
I20260928 16:44:43.978515 3190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:44.015657 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:44.136834 3202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:44.137161 3202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:44.137209 3202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:44.141476 3202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:44.141557 3202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:44.141625 3202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:44.147789 3202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:44.149483 3202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:44.150640 3202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:44.161185 3210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.161237 3207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.161433 3202 server_base.cc:1034] running on GCE node
W20260928 16:44:44.161237 3208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.162037 3202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:44.162734 3202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:44.163923 3202 hybrid_clock.cc:648] HybridClock initialized: now 1790613884163904 us; error 33 us; skew 500 ppm
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:44.166841 3202 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:44.167644 3202 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:44.167768 3202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:44.168015 3202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:44.170379 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "5097b2e6244fe93af3a6437b030486f9"
server_key_iv: "4640f0ad260f688c60c769dbfdb1f638"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.170981 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "5097b2e6244fe93af3a6437b030486f9"
server_key_iv: "4640f0ad260f688c60c769dbfdb1f638"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.174854 3202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:44.177590 3216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.178897 3202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:44.179013 3202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "5097b2e6244fe93af3a6437b030486f9"
server_key_iv: "4640f0ad260f688c60c769dbfdb1f638"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.179162 3202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:44.192962 3202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:44.196576 3202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:44.196816 3202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:44.197459 3202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:44.198334 3202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:44.198382 3202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.198424 3202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:44.198441 3202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.208674 3202 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:38725
I20260928 16:44:44.208696 3329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:38725 every 8 connection(s)
I20260928 16:44:44.209357 3202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:44.214807 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3202
I20260928 16:44:44.215072 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:44.215350 31504 external_mini_cluster.cc:1428] Setting key 7abd98cc0e65c310d98c6951292eacd3
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:44.227087 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.211525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:57819 (local address 127.30.196.62:38515)
0928 16:44:44.211720 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.211724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.212896 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 16:44:44.214516 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.214519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.214594 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.214703 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.216337 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.217173 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.218421 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.218879 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.218989 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.222377 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.222381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.222385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:44.222421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.224561 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.225322 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.225327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.225329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.225436 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.225837 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.225841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.225843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.226088 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.226203 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.226834 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.226952 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:44:44.228075 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.212000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38515 (local address 127.30.196.1:57819)
0928 16:44:44.212535 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.212640 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.214242 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.214907 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.214914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.215413 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:44.216162 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.216181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.217326 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.217329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.218269 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.218279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.219089 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.219095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.219346 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.220155 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:44.220181 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:44.222148 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:44.224729 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.224735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.224752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.225178 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.225549 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.225551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.225553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.225714 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.226222 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:44.226228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:44.226556 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.226745 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.227090 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:44.229606 3330 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38515
I20260928 16:44:44.229918 3330 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:44.230510 3330 heartbeater.cc:511] Master 127.30.196.62:38515 requested a full tablet report, sending...
I20260928 16:44:44.232191 3140 ts_manager.cc:194] Registered new tserver with Master: b55e6bb8ea244851ba2f7c8cd593e7ca (127.30.196.1:38725)
I20260928 16:44:44.233922 3140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:57819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:44.248502 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:44.366726 3335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:44.367008 3335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:44.367055 3335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:44.371351 3335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:44.371425 3335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:44.371487 3335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:44.377041 3335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:44.378142 3335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:44.379154 3335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:44.389245 3341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.389231 3340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.389233 3343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.389734 3335 server_base.cc:1034] running on GCE node
I20260928 16:44:44.390254 3335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:44.391027 3335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:44.392252 3335 hybrid_clock.cc:648] HybridClock initialized: now 1790613884392237 us; error 30 us; skew 500 ppm
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:44.395294 3335 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:44.396153 3335 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:44.396224 3335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:44.396404 3335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:44.398452 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "886739b32c0c42f992a439833c3488db"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ce3b7db48d05d1526eeab51570a5fea5"
server_key_iv: "23a3afb813fcdbbc714eed6ba097af90"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.398969 3335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "886739b32c0c42f992a439833c3488db"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ce3b7db48d05d1526eeab51570a5fea5"
server_key_iv: "23a3afb813fcdbbc714eed6ba097af90"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.402823 3335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:44.405368 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.406651 3335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:44:44.406751 3335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "886739b32c0c42f992a439833c3488db"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ce3b7db48d05d1526eeab51570a5fea5"
server_key_iv: "23a3afb813fcdbbc714eed6ba097af90"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.406845 3335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:44.422545 3335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:44.425927 3335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:44.426095 3335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:44.426648 3335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:44.427479 3335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:44.427528 3335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.427570 3335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:44.427593 3335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.437057 3335 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:37957
I20260928 16:44:44.437074 3462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:37957 every 8 connection(s)
I20260928 16:44:44.437727 3335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:44.446178 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3335
I20260928 16:44:44.446327 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:44.446642 31504 external_mini_cluster.cc:1428] Setting key e411579ea72ffb7844c09f3f5a8fd48f
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:44.465878 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.439503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:57405 (local address 127.30.196.62:38515)
0928 16:44:44.439686 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.439690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.440629 (+ 939us) server_negotiation.cc:407] Connection header received
0928 16:44:44.442070 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.442074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.442136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.442252 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.443467 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.444072 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.445238 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.445473 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.445599 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.461297 (+ 15698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.461304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.461311 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:44.461353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.463112 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.463904 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.463911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.463916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.464019 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.464472 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.464475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.464477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.464757 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.464873 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.465435 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.465607 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:44.466601 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.439971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38515 (local address 127.30.196.2:57405)
0928 16:44:44.440450 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.440493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.441833 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.442431 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.442438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.442811 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:44.443308 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.443322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.444207 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.444216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.445108 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.445117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.445637 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.445641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.445850 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.457000 (+ 11150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:44.457024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:44.461040 (+ 4016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:44.463288 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.463294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.463309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.463782 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.464159 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.464163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.464167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.464351 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.464871 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:44.464877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:44.465149 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.465317 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.465613 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260928 16:44:44.467970 3463 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38515
I20260928 16:44:44.468269 3463 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:44.468878 3463 heartbeater.cc:511] Master 127.30.196.62:38515 requested a full tablet report, sending...
I20260928 16:44:44.469980 3140 ts_manager.cc:194] Registered new tserver with Master: 886739b32c0c42f992a439833c3488db (127.30.196.2:37957)
I20260928 16:44:44.470638 3140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:57405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:44.479421 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:44.597783 3468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:44.598037 3468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:44.598122 3468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:44.602272 3468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:44.602342 3468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:44.602445 3468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:44.608026 3468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:44.609467 3468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:44.610579 3468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:44.620613 3476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.620613 3474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.620750 3468 server_base.cc:1034] running on GCE node
W20260928 16:44:44.620957 3473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.621431 3468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:44.622185 3468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:44.623374 3468 hybrid_clock.cc:648] HybridClock initialized: now 1790613884623360 us; error 27 us; skew 500 ppm
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:44.626413 3468 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:44.627336 3468 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:44.627403 3468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:44.627633 3468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:44.630046 3468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "7e239e7123874feba4871a13532ad308"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ee32e38c38b21da67bbb5a1af01643a7"
server_key_iv: "155d64cd7cac0a87b97eb8addc4be7a9"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.630750 3468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "7e239e7123874feba4871a13532ad308"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ee32e38c38b21da67bbb5a1af01643a7"
server_key_iv: "155d64cd7cac0a87b97eb8addc4be7a9"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.634989 3468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:44.637830 3482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.639183 3468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:44.639339 3468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "7e239e7123874feba4871a13532ad308"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-5rg1"
server_key: "ee32e38c38b21da67bbb5a1af01643a7"
server_key_iv: "155d64cd7cac0a87b97eb8addc4be7a9"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.639484 3468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:44.665176 3468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:44.668793 3468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:44.669034 3468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:44.669655 3468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:44.670537 3468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:44.670586 3468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.670629 3468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:44.670668 3468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.680475 3468 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:37481
I20260928 16:44:44.680495 3595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:37481 every 8 connection(s)
I20260928 16:44:44.681106 3468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:44.686930 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3468
I20260928 16:44:44.687084 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:44.687393 31504 external_mini_cluster.cc:1428] Setting key c418c9a61298378c51917030da3c698d
Sep 28 16:44:44 dist-test-slave-5rg1 krb5kdc[3095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613884, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:44.696246 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.683031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:40601 (local address 127.30.196.62:38515)
0928 16:44:44.683206 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.683209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.684241 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 16:44:44.685617 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.685621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.685696 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.685805 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.687044 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.687785 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.689539 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.689871 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.689955 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.692643 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.692646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.692648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:44.692674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.694074 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.694648 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.694651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.694653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.694737 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.695085 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.695087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.695088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.695281 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.695363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.695982 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.696112 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:44:44.697142 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.683473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38515 (local address 127.30.196.3:40601)
0928 16:44:44.684016 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.684062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.685382 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.685982 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.685989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.686380 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:44.686879 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.686893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.687965 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.687969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.689396 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.689414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.690014 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.690017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.690173 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.690786 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:44.690803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:44.692457 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:44.694211 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.694216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.694230 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.694530 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.694836 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:44.694839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:44.694840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:44.694983 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:44.695377 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:44.695383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:44.695656 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.695914 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.696203 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":180,"threads_started":1}
I20260928 16:44:44.698350 3596 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38515
I20260928 16:44:44.698576 3596 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:44.699000 3596 heartbeater.cc:511] Master 127.30.196.62:38515 requested a full tablet report, sending...
I20260928 16:44:44.700102 3140 ts_manager.cc:194] Registered new tserver with Master: 7e239e7123874feba4871a13532ad308 (127.30.196.3:37481)
I20260928 16:44:44.700863 3140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:40601
I20260928 16:44:44.701519 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:44.711274 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.703459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.30.196.62:38515)
0928 16:44:44.703584 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.703588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.703760 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:44:44.703864 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.703866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.703904 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.703973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.704649 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.705146 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.705965 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.706203 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.706297 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.707814 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.707817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.707819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:44.707847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.709300 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.709750 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.709753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.709755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.709848 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:44.710205 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:44.710208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:44.710209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:44.710523 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.710642 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.710946 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.711082 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:44.714557 3139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:44.717721 3139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:44.733002 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.726684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.30.196.1:38725)
0928 16:44:44.727994 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.728001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.728025 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:44:44.728108 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.728113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.728337 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.728877 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:44.729031 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.729805 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.731341 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732232 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732341 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.732496 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.732726 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.732829 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":85,"threads_started":1}
I20260928 16:44:44.732985 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.726512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38725 (local address 127.0.0.1:54424)
0928 16:44:44.727071 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.727126 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.727270 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.728520 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.728526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.728552 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:44.728712 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.728718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.729923 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.729930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.731196 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.731207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732376 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.732516 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.732556 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.732676 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.732805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
I20260928 16:44:44.733325 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.727423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.30.196.3:37481)
0928 16:44:44.727878 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.727890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.728065 (+ 175us) server_negotiation.cc:407] Connection header received
0928 16:44:44.728199 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.728204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.728403 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.729045 (+ 642us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:44.729208 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.730322 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.731802 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732397 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732506 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.732664 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.733055 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.733161 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260928 16:44:44.733516 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.726704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37481 (local address 127.0.0.1:38954)
0928 16:44:44.727954 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.727970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.728075 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.728741 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.728744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.728763 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:44.728885 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.728892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.730477 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.730480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.731698 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.731708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732725 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.732810 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.732829 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.732897 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.732952 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":60,"thread_start_us":105,"threads_started":1}
I20260928 16:44:44.733824 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.727129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37957 (local address 127.0.0.1:40534)
0928 16:44:44.728275 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.728287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.728359 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.728881 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.728884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.728912 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:44.729019 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.729025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.730501 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.730504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.731653 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.731724 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732931 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.733040 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.733546 (+ 506us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.733632 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.733704 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260928 16:44:44.734439 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.727367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.30.196.2:37957)
0928 16:44:44.727737 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.727749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.728451 (+ 702us) server_negotiation.cc:407] Connection header received
0928 16:44:44.728562 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.728567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.728742 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.728890 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:44.729149 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.730322 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.731849 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.732621 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.732747 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.732868 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.734047 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.734144 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260928 16:44:44.736253 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc (DEFAULT_TABLE table=test-table [id=182d23dfa5d44ad1ab74852744b445bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:44.736287 3530 tablet_service.cc:1467] Processing CreateTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc (DEFAULT_TABLE table=test-table [id=182d23dfa5d44ad1ab74852744b445bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:44.737238 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35533b8faa3d49caaa583c5f2e1e88dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:44.737315 3530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35533b8faa3d49caaa583c5f2e1e88dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:44.737061 3397 tablet_service.cc:1467] Processing CreateTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc (DEFAULT_TABLE table=test-table [id=182d23dfa5d44ad1ab74852744b445bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:44.737974 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35533b8faa3d49caaa583c5f2e1e88dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:44.744460 3613 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Bootstrap starting.
I20260928 16:44:44.744982 3615 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Bootstrap starting.
I20260928 16:44:44.745227 3614 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Bootstrap starting.
I20260928 16:44:44.747231 3615 tablet_bootstrap.cc:654] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:44.747231 3613 tablet_bootstrap.cc:654] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:44.747799 3614 tablet_bootstrap.cc:654] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:44.748262 3613 log.cc:824] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:44.748593 3615 log.cc:824] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:44.748642 3614 log.cc:824] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:44.750810 3615 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: No bootstrap required, opened a new log
I20260928 16:44:44.750818 3614 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: No bootstrap required, opened a new log
I20260928 16:44:44.750818 3613 tablet_bootstrap.cc:492] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: No bootstrap required, opened a new log
I20260928 16:44:44.751025 3614 ts_tablet_manager.cc:1397] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:44:44.751026 3613 ts_tablet_manager.cc:1397] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:44:44.751037 3615 ts_tablet_manager.cc:1397] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 16:44:44.754139 3615 raft_consensus.cc:348] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.754441 3615 raft_consensus.cc:374] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:44.754504 3615 raft_consensus.cc:729] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e239e7123874feba4871a13532ad308, State: Initialized, Role: FOLLOWER
I20260928 16:44:44.754978 3615 consensus_queue.cc:260] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.755067 3614 raft_consensus.cc:348] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.755366 3614 raft_consensus.cc:374] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:44.755427 3614 raft_consensus.cc:729] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886739b32c0c42f992a439833c3488db, State: Initialized, Role: FOLLOWER
I20260928 16:44:44.755515 3613 raft_consensus.cc:348] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.755779 3613 raft_consensus.cc:374] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:44.755829 3613 raft_consensus.cc:729] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55e6bb8ea244851ba2f7c8cd593e7ca, State: Initialized, Role: FOLLOWER
I20260928 16:44:44.755859 3596 heartbeater.cc:503] Master 127.30.196.62:38515 was elected leader, sending a full tablet report...
I20260928 16:44:44.755936 3614 consensus_queue.cc:260] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.756134 3615 ts_tablet_manager.cc:1428] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:44.756263 3613 consensus_queue.cc:260] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.756723 3463 heartbeater.cc:503] Master 127.30.196.62:38515 was elected leader, sending a full tablet report...
I20260928 16:44:44.757153 3613 ts_tablet_manager.cc:1428] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:44:44.757606 3330 heartbeater.cc:503] Master 127.30.196.62:38515 was elected leader, sending a full tablet report...
I20260928 16:44:44.757743 3614 ts_tablet_manager.cc:1428] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 16:44:44.932045 3597 tablet.cc:2369] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:44.938813 3464 tablet.cc:2369] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:44.960784 3331 tablet.cc:2369] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:44.966725 3620 raft_consensus.cc:482] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:44.966954 3620 raft_consensus.cc:504] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.968282 3620 leader_election.cc:288] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b55e6bb8ea244851ba2f7c8cd593e7ca (127.30.196.1:38725), 7e239e7123874feba4871a13532ad308 (127.30.196.3:37481)
I20260928 16:44:44.972396 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.968853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33119 (local address 127.30.196.3:37481)
0928 16:44:44.969008 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.969014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.969031 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:44.969136 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.969139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.969180 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.969272 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:44.969641 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.970128 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.971338 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.971781 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.971879 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.971921 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.972174 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.972263 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:44.972431 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.968512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37481 (local address 127.30.196.2:33119)
0928 16:44:44.968872 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.968888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.969011 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.969289 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.969292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.969336 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:44.969489 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.969494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.970246 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.970252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.971206 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.971215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.971906 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.971910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.972031 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.972060 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.972146 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.972217 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":44,"spinlock_wait_cycles":5120,"thread_start_us":68,"threads_started":1}
I20260928 16:44:44.972759 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.968512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38725 (local address 127.30.196.2:59743)
0928 16:44:44.968767 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.968783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.968920 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.969287 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.969290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.969333 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:44.969479 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.969484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.970308 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.970310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.971204 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:44.971213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.972049 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.972053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:44.972184 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.972208 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.972284 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.972352 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":31,"spinlock_wait_cycles":36864,"thread_start_us":91,"threads_started":1}
I20260928 16:44:44.973166 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.968633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:59743 (local address 127.30.196.1:38725)
0928 16:44:44.968767 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.968773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.968863 (+ 90us) server_negotiation.cc:407] Connection header received
0928 16:44:44.969105 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.969111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.969177 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.969270 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:44.969595 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.970163 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.971345 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:44.971909 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:44.972044 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:44.972096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.972322 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.972397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:44.973496 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35533b8faa3d49caaa583c5f2e1e88dc" candidate_uuid: "886739b32c0c42f992a439833c3488db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e239e7123874feba4871a13532ad308" is_pre_election: true
I20260928 16:44:44.973845 3550 raft_consensus.cc:2454] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886739b32c0c42f992a439833c3488db in term 0.
I20260928 16:44:44.973855 3284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35533b8faa3d49caaa583c5f2e1e88dc" candidate_uuid: "886739b32c0c42f992a439833c3488db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" is_pre_election: true
I20260928 16:44:44.974126 3284 raft_consensus.cc:2454] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886739b32c0c42f992a439833c3488db in term 0.
I20260928 16:44:44.974419 3353 leader_election.cc:302] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e239e7123874feba4871a13532ad308, 886739b32c0c42f992a439833c3488db; no voters:
I20260928 16:44:44.974757 3620 raft_consensus.cc:2781] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:44.974845 3620 raft_consensus.cc:482] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:44.974900 3620 raft_consensus.cc:3037] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:44.976145 3620 raft_consensus.cc:504] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.976591 3620 leader_election.cc:288] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [CANDIDATE]: Term 1 election: Requested vote from peers b55e6bb8ea244851ba2f7c8cd593e7ca (127.30.196.1:38725), 7e239e7123874feba4871a13532ad308 (127.30.196.3:37481)
I20260928 16:44:44.976958 3284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35533b8faa3d49caaa583c5f2e1e88dc" candidate_uuid: "886739b32c0c42f992a439833c3488db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca"
I20260928 16:44:44.977141 3550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35533b8faa3d49caaa583c5f2e1e88dc" candidate_uuid: "886739b32c0c42f992a439833c3488db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e239e7123874feba4871a13532ad308"
I20260928 16:44:44.977136 3284 raft_consensus.cc:3037] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:44.977290 3550 raft_consensus.cc:3037] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:44.978322 3284 raft_consensus.cc:2454] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886739b32c0c42f992a439833c3488db in term 1.
I20260928 16:44:44.978500 3550 raft_consensus.cc:2454] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886739b32c0c42f992a439833c3488db in term 1.
I20260928 16:44:44.978852 3352 leader_election.cc:302] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7e239e7123874feba4871a13532ad308, 886739b32c0c42f992a439833c3488db, b55e6bb8ea244851ba2f7c8cd593e7ca; no voters:
I20260928 16:44:44.979079 3620 raft_consensus.cc:2781] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:44.979358 3620 raft_consensus.cc:686] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 1 LEADER]: Becoming Leader. State: Replica: 886739b32c0c42f992a439833c3488db, State: Running, Role: LEADER
I20260928 16:44:44.979825 3620 consensus_queue.cc:237] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } }
I20260928 16:44:44.983245 3137 catalog_manager.cc:5495] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db reported cstate change: term changed from 0 to 1, leader changed from <none> to 886739b32c0c42f992a439833c3488db (127.30.196.2). New cstate: current_term: 1 leader_uuid: "886739b32c0c42f992a439833c3488db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886739b32c0c42f992a439833c3488db" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37957 } health_report { overall_health: HEALTHY } } }
I20260928 16:44:45.029759 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.026053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.30.196.2:37957)
0928 16:44:45.026198 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.026202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.026216 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:45.026296 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.026298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.026354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.026451 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:45.026718 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:45.027251 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:45.028137 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:45.028371 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:45.028473 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:45.028815 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:45.029340 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:45.029454 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.029565 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.029633 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:44:45.036468 3550 raft_consensus.cc:1261] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 1 FOLLOWER]: Refusing update from remote peer 886739b32c0c42f992a439833c3488db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:45.036468 3284 raft_consensus.cc:1261] T 35533b8faa3d49caaa583c5f2e1e88dc P b55e6bb8ea244851ba2f7c8cd593e7ca [term 1 FOLLOWER]: Refusing update from remote peer 886739b32c0c42f992a439833c3488db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:45.037346 3624 consensus_queue.cc:1035] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b55e6bb8ea244851ba2f7c8cd593e7ca" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:45.037544 3620 consensus_queue.cc:1035] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e239e7123874feba4871a13532ad308" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:45.046231 3629 mvcc.cc:205] Tried to move back new op lower bound from 7334354473101127680 to 7334354472880386048. Current Snapshot: MvccSnapshot[applied={T|T < 7334354473101127680}]
I20260928 16:44:45.055125 3140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:45.055347 3140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:45.058148 3140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae36c0776af44309873c06ea08b42f12: Sending DeleteTablet for 3 replicas of tablet 35533b8faa3d49caaa583c5f2e1e88dc
I20260928 16:44:45.059094 3530 tablet_service.cc:1514] Processing DeleteTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:45 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260928 16:44:45.059101 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:45 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:54424
I20260928 16:44:45.059337 3397 tablet_service.cc:1514] Processing DeleteTablet for tablet 35533b8faa3d49caaa583c5f2e1e88dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:45 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:40534
I20260928 16:44:45.059919 3638 tablet_replica.cc:330] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: stopping tablet replica
I20260928 16:44:45.059909 3639 tablet_replica.cc:330] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: stopping tablet replica
I20260928 16:44:45.060283 3639 raft_consensus.cc:2229] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:45.060355 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3202
I20260928 16:44:45.062518 3638 raft_consensus.cc:2229] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:45.062927 3638 raft_consensus.cc:2258] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:45.063150 3639 raft_consensus.cc:2258] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:45.064812 3638 ts_tablet_manager.cc:1905] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:45.064811 3639 ts_tablet_manager.cc:1905] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:45.067754 3639 ts_tablet_manager.cc:1918] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:45.067754 3638 ts_tablet_manager.cc:1918] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:45.067865 3638 log.cc:1196] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/35533b8faa3d49caaa583c5f2e1e88dc
I20260928 16:44:45.068030 3639 log.cc:1196] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/35533b8faa3d49caaa583c5f2e1e88dc
I20260928 16:44:45.068249 3638 ts_tablet_manager.cc:1939] T 35533b8faa3d49caaa583c5f2e1e88dc P 7e239e7123874feba4871a13532ad308: Deleting consensus metadata
I20260928 16:44:45.068498 3639 ts_tablet_manager.cc:1939] T 35533b8faa3d49caaa583c5f2e1e88dc P 886739b32c0c42f992a439833c3488db: Deleting consensus metadata
W20260928 16:44:45.068770 3126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:38725 (error 108)
I20260928 16:44:45.068943 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3335
I20260928 16:44:45.071285 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.070895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:38725 (local address 127.0.0.1:54440)
0928 16:44:45.071051 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.071173 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:38725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 16:44:45.071287 3127 catalog_manager.cc:4841] TS 7e239e7123874feba4871a13532ad308 (127.30.196.3:37481): tablet 35533b8faa3d49caaa583c5f2e1e88dc (table test-table [id=182d23dfa5d44ad1ab74852744b445bf]) successfully deleted
W20260928 16:44:45.071506 3126 catalog_manager.cc:4584] TS b55e6bb8ea244851ba2f7c8cd593e7ca (127.30.196.1:38725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35533b8faa3d49caaa583c5f2e1e88dc: Network error: Client connection negotiation failed: client connection to 127.30.196.1:38725: connect: Connection refused (error 111)
I20260928 16:44:45.075798 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3468
I20260928 16:44:45.077605 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.077208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37957 (local address 127.0.0.1:40546)
0928 16:44:45.077415 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.077494 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 16:44:45.082091 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3109
2026-09-28T16:44:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2538 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T16:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:45Z Disabled control of system clock
I20260928 16:44:45.115216 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46461
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:44:45.234175 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:45.234488 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:45.234531 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:45.239027 3644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:45.239099 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:45.239115 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:45.239140 3644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:45.239157 3644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:45.245383 3644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:45.246621 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:45.248226 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:45.257597 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.257663 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.257597 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.258287 3644 server_base.cc:1034] running on GCE node
I20260928 16:44:45.258924 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:45.260332 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:45.261507 3644 hybrid_clock.cc:648] HybridClock initialized: now 1790613885261495 us; error 31 us; skew 500 ppm
I20260928 16:44:45.263839 3644 webserver.cc:460] Webserver started at http://127.30.196.62:34629/ using document root <none> and password file <none>
I20260928 16:44:45.264451 3644 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:45.264519 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:45.264777 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:45.266913 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "6b82ea84c3d84034bf8766a539f443dc"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "e32b4c9434505fc3a95eb54409360b56"
server_key_iv: "138bccbcbcae05e633d200db3cd1800e"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.267493 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "6b82ea84c3d84034bf8766a539f443dc"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "e32b4c9434505fc3a95eb54409360b56"
server_key_iv: "138bccbcbcae05e633d200db3cd1800e"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.271303 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:44:45.273929 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.275336 3644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:45.275485 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "6b82ea84c3d84034bf8766a539f443dc"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "e32b4c9434505fc3a95eb54409360b56"
server_key_iv: "138bccbcbcae05e633d200db3cd1800e"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.275624 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:45.280387 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:45.281051 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:45.281246 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:45.288539 3644 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:46461
I20260928 16:44:45.288555 3710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:46461 every 8 connection(s)
I20260928 16:44:45.289526 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:45.291927 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3644
I20260928 16:44:45.292053 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:45.292376 31504 external_mini_cluster.cc:1428] Setting key c90166be1e7a75e983749f6e231c217c
I20260928 16:44:45.294687 3711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:45.298139 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.293817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.30.196.62:46461)
0928 16:44:45.294501 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.294515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.294559 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:44:45.295280 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.295298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.295652 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.296066 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.296837 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.296842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.296852 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.296862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.296951 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.296968 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.297134 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.297210 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.297454 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":122,"threads_started":1}
I20260928 16:44:45.301384 3711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: Bootstrap starting.
I20260928 16:44:45.304148 3711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:45.305102 3711 log.cc:824] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:45.307333 3711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: No bootstrap required, opened a new log
I20260928 16:44:45.311414 3711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } }
I20260928 16:44:45.311740 3711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:45.311823 3711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b82ea84c3d84034bf8766a539f443dc, State: Initialized, Role: FOLLOWER
I20260928 16:44:45.312379 3711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } }
I20260928 16:44:45.312556 3711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:45.312633 3711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:45.312729 3711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:45.314116 3711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } }
I20260928 16:44:45.314556 3711 leader_election.cc:302] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b82ea84c3d84034bf8766a539f443dc; no voters:
I20260928 16:44:45.315029 3711 leader_election.cc:288] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:45.315352 3716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:45.315564 3716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [term 1 LEADER]: Becoming Leader. State: Replica: 6b82ea84c3d84034bf8766a539f443dc, State: Running, Role: LEADER
I20260928 16:44:45.315917 3716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } }
I20260928 16:44:45.316509 3711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:45.317689 3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b82ea84c3d84034bf8766a539f443dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } } }
I20260928 16:44:45.317830 3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:45.318215 3725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:45.318169 3718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b82ea84c3d84034bf8766a539f443dc. Latest consensus state: current_term: 1 leader_uuid: "6b82ea84c3d84034bf8766a539f443dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b82ea84c3d84034bf8766a539f443dc" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46461 } } }
I20260928 16:44:45.318818 3718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:45.321657 3725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:45.327450 3725 catalog_manager.cc:1347] Generated new cluster ID: f27adfdc953c4946a65e5a961bd67edf
I20260928 16:44:45.327545 3725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:45.337100 3725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:45.337472 3725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:45.344516 3725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: Generated new TSK 0
I20260928 16:44:45.345289 3725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:45.352553 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--builtin_ntp_servers=127.30.196.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:44:45.471375 3735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:45.471647 3735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:45.471762 3735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:45.475970 3735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:45.476042 3735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:45.476101 3735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:45.481352 3735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:45.482645 3735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:45.484122 3735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:45.494087 3740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.494109 3743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.494086 3741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.494313 3735 server_base.cc:1034] running on GCE node
I20260928 16:44:45.494973 3735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:45.495591 3735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:45.496776 3735 hybrid_clock.cc:648] HybridClock initialized: now 1790613885496755 us; error 34 us; skew 500 ppm
I20260928 16:44:45.498724 3735 webserver.cc:460] Webserver started at http://127.30.196.1:43241/ using document root <none> and password file <none>
I20260928 16:44:45.499354 3735 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:45.499426 3735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:45.499704 3735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:45.501924 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "25145ac0f28b4d4d95289d8106d19124"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "b53312e1112b5fa873f9814da3139e7f"
server_key_iv: "0c74ed29a49374663a671d40f4fec2a8"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.502533 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "25145ac0f28b4d4d95289d8106d19124"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "b53312e1112b5fa873f9814da3139e7f"
server_key_iv: "0c74ed29a49374663a671d40f4fec2a8"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.506482 3735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:45.509358 3749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.510725 3735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:45.510869 3735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "25145ac0f28b4d4d95289d8106d19124"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "b53312e1112b5fa873f9814da3139e7f"
server_key_iv: "0c74ed29a49374663a671d40f4fec2a8"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.511003 3735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:45.516273 3735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:45.517014 3735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:45.517233 3735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:45.517917 3735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:45.518909 3735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:45.518959 3735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.519028 3735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:45.519059 3735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.528554 3735 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:36379
I20260928 16:44:45.528573 3862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:36379 every 8 connection(s)
I20260928 16:44:45.529652 3735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:45.537177 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.532123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:49721 (local address 127.30.196.62:46461)
0928 16:44:45.532318 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.532322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.533595 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 16:44:45.535004 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.535007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.535135 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.535240 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.536641 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.536645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.536648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.536660 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.536687 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.536702 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.536813 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.536969 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.537020 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:44:45.538123 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.532562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46461 (local address 127.30.196.1:49721)
0928 16:44:45.533422 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.533469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.534774 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.535434 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.535441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.535584 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.536327 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.536329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.536373 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.536468 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.536805 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.536814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.536821 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.537018 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.537277 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
I20260928 16:44:45.539528 3863 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46461
I20260928 16:44:45.539852 3863 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:45.539961 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3735
I20260928 16:44:45.540099 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:45.540396 31504 external_mini_cluster.cc:1428] Setting key 9f1938cb3b01758259d3ab678939b455
I20260928 16:44:45.540472 3863 heartbeater.cc:511] Master 127.30.196.62:46461 requested a full tablet report, sending...
I20260928 16:44:45.542448 3674 ts_manager.cc:194] Registered new tserver with Master: 25145ac0f28b4d4d95289d8106d19124 (127.30.196.1:36379)
I20260928 16:44:45.542610 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--builtin_ntp_servers=127.30.196.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:44:45.658861 3866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:45.659168 3866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:45.659210 3866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:45.663496 3866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:45.663585 3866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:45.663646 3866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:45.668735 3866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:45.670003 3866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:45.671406 3866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:45.682370 3872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.682390 3871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.682446 3866 server_base.cc:1034] running on GCE node
W20260928 16:44:45.682386 3874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.683102 3866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:45.683856 3866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:45.685201 3866 hybrid_clock.cc:648] HybridClock initialized: now 1790613885685157 us; error 52 us; skew 500 ppm
I20260928 16:44:45.687237 3866 webserver.cc:460] Webserver started at http://127.30.196.2:44603/ using document root <none> and password file <none>
I20260928 16:44:45.687947 3866 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:45.688042 3866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:45.688266 3866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:45.690660 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "b6be26657d4142cab8b1ac442aa9fec6"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "730f528318fc21ffd9c0551bb4f07978"
server_key_iv: "b012705684cf984fcd40ed52f33093c0"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.691247 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "b6be26657d4142cab8b1ac442aa9fec6"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "730f528318fc21ffd9c0551bb4f07978"
server_key_iv: "b012705684cf984fcd40ed52f33093c0"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.695400 3866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:45.698415 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.700052 3866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:45.700207 3866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "b6be26657d4142cab8b1ac442aa9fec6"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "730f528318fc21ffd9c0551bb4f07978"
server_key_iv: "b012705684cf984fcd40ed52f33093c0"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.700364 3866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:45.705560 3866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:45.706226 3866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:45.706440 3866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:45.707018 3866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:45.708055 3866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:45.708127 3866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.708200 3866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:45.708251 3866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.718029 3866 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:37875
I20260928 16:44:45.718091 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:37875 every 8 connection(s)
I20260928 16:44:45.719180 3866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:45.720130 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3866
I20260928 16:44:45.720257 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:45.720551 31504 external_mini_cluster.cc:1428] Setting key 592578a932d60bd5f3ea7f319eda5352
I20260928 16:44:45.724416 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--builtin_ntp_servers=127.30.196.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:44:45.726200 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.721349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:37105 (local address 127.30.196.62:46461)
0928 16:44:45.721526 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.721531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.722751 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 16:44:45.724247 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.724251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.724302 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.724405 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.725655 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.725662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.725668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.725687 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.725716 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.725729 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.725843 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.726046 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.726089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:45.727537 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.721882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46461 (local address 127.30.196.2:37105)
0928 16:44:45.722593 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.722636 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.724020 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.724620 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.724633 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.724748 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.725377 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.725378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.725423 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.725495 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.725854 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.725866 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.725875 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.726082 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.726363 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260928 16:44:45.729140 3994 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46461
I20260928 16:44:45.729426 3994 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:45.729895 3994 heartbeater.cc:511] Master 127.30.196.62:46461 requested a full tablet report, sending...
I20260928 16:44:45.731153 3674 ts_manager.cc:194] Registered new tserver with Master: b6be26657d4142cab8b1ac442aa9fec6 (127.30.196.2:37875)
W20260928 16:44:45.842397 3997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:45.842691 3997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:45.842733 3997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:45.846779 3997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:45.846838 3997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:45.846896 3997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:45.851732 3997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:45.852805 3997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:45.854135 3997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:45.864260 4005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.864260 4003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.864274 4002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.864497 3997 server_base.cc:1034] running on GCE node
I20260928 16:44:45.865003 3997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:45.865732 3997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:45.866935 3997 hybrid_clock.cc:648] HybridClock initialized: now 1790613885866905 us; error 45 us; skew 500 ppm
I20260928 16:44:45.869196 3997 webserver.cc:460] Webserver started at http://127.30.196.3:39417/ using document root <none> and password file <none>
I20260928 16:44:45.869849 3997 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:45.869930 3997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:45.870172 3997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:45.872766 3997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "ecf1e20fb0204f3bac1b5532d6649093"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "3d2a6249c330d9d49cb630866cb6e6a0"
server_key_iv: "71af3b48e271b3f82de61d90164b202b"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.873498 3997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "ecf1e20fb0204f3bac1b5532d6649093"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "3d2a6249c330d9d49cb630866cb6e6a0"
server_key_iv: "71af3b48e271b3f82de61d90164b202b"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.877544 3997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:45.880432 4011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.881796 3997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:45.881951 3997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "ecf1e20fb0204f3bac1b5532d6649093"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-5rg1"
server_key: "3d2a6249c330d9d49cb630866cb6e6a0"
server_key_iv: "71af3b48e271b3f82de61d90164b202b"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.882082 3997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:45.886768 3997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:45.887357 3997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:45.887544 3997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:45.888156 3997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:45.889219 3997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:45.889281 3997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.889320 3997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:45.889336 3997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.899293 3997 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:40395
I20260928 16:44:45.899314 4124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:40395 every 8 connection(s)
I20260928 16:44:45.900355 3997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:45.902462 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 3997
I20260928 16:44:45.902616 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:45.902925 31504 external_mini_cluster.cc:1428] Setting key 17004863e91af3feb69c1aac469ccc8a
I20260928 16:44:45.907508 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.903352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:53021 (local address 127.30.196.62:46461)
0928 16:44:45.903496 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.903504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.904183 (+ 679us) server_negotiation.cc:407] Connection header received
0928 16:44:45.905836 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.905838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.905878 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.905954 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.907147 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.907150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.907152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.907162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.907180 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.907187 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.907256 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.907356 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.907394 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 16:44:45.908483 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.902919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46461 (local address 127.30.196.3:53021)
0928 16:44:45.904028 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.904071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.905628 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.906098 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.906104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.906195 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.906882 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.906884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.906928 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.906998 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.907245 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.907249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.907253 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.907376 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.907615 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":156,"threads_started":1}
I20260928 16:44:45.909710 4125 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46461
I20260928 16:44:45.909924 4125 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:45.910305 4125 heartbeater.cc:511] Master 127.30.196.62:46461 requested a full tablet report, sending...
I20260928 16:44:45.911366 3674 ts_manager.cc:194] Registered new tserver with Master: ecf1e20fb0204f3bac1b5532d6649093 (127.30.196.3:40395)
I20260928 16:44:45.917192 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:45.921186 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.919323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.30.196.62:46461)
0928 16:44:45.919498 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.919503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.919638 (+ 135us) server_negotiation.cc:407] Connection header received
0928 16:44:45.919838 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.919841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.919888 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.920022 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.920770 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.920773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.920776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.920788 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.920808 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.920819 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.920914 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.920996 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.921040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:45.924340 3674 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:45.927498 3674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:45.941283 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.937361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.30.196.2:37875)
0928 16:44:45.937963 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.937982 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.938344 (+ 362us) server_negotiation.cc:407] Connection header received
0928 16:44:45.938407 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.938413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.938636 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.938775 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.940585 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.940594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.940607 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.940624 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.940675 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.940718 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.940841 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.941024 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.941119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":110,"threads_started":1}
I20260928 16:44:45.941473 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.938388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.30.196.3:40395)
0928 16:44:45.938930 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.938944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.938983 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:44:45.939047 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.939055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.939270 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.939415 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.940799 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.940805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.940815 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.940834 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.940879 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.940914 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.941013 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.941256 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.941320 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":186,"threads_started":1}
I20260928 16:44:45.941473 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.938457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.30.196.1:36379)
0928 16:44:45.939433 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.939446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.939468 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:45.939525 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.939529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.939651 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.939846 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.940569 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.940576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.940603 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.940615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.940651 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.940673 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.941205 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.941254 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.941320 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":120,"threads_started":1}
I20260928 16:44:45.941741 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.937234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37875 (local address 127.0.0.1:46434)
0928 16:44:45.938029 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.938143 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.938236 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.938899 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.938905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.938984 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.940113 (+ 1129us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.940117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.940141 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.940220 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.940807 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.940817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.940829 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.940979 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.941024 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"spinlock_wait_cycles":4864,"thread_start_us":90,"threads_started":1}
I20260928 16:44:45.941772 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.937342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:36379 (local address 127.0.0.1:37326)
0928 16:44:45.938091 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.938109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.938236 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.939835 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.939838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.939856 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.940381 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.940385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.940400 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.940430 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.940813 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.940829 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.940834 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.940931 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.941013 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"spinlock_wait_cycles":7552,"thread_start_us":150,"threads_started":1}
I20260928 16:44:45.941994 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.937994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:40395 (local address 127.0.0.1:33458)
0928 16:44:45.938423 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.938435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.938495 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.939782 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.939788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.939811 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.940610 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.940612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.940624 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.940649 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.941065 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.941068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.941072 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.941155 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.941187 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":126,"threads_started":1}
I20260928 16:44:45.945120 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 65fea68874d64e919043e1dae0f941de (DEFAULT_TABLE table=test-table [id=297652bd83824b4086937aac06b13f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:45.945120 3928 tablet_service.cc:1467] Processing CreateTablet for tablet 65fea68874d64e919043e1dae0f941de (DEFAULT_TABLE table=test-table [id=297652bd83824b4086937aac06b13f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:45.945125 4059 tablet_service.cc:1467] Processing CreateTablet for tablet 65fea68874d64e919043e1dae0f941de (DEFAULT_TABLE table=test-table [id=297652bd83824b4086937aac06b13f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:45.946342 4059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fea68874d64e919043e1dae0f941de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:45.946347 3928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fea68874d64e919043e1dae0f941de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:45.946347 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fea68874d64e919043e1dae0f941de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:45.953699 4142 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Bootstrap starting.
I20260928 16:44:45.954689 4144 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Bootstrap starting.
I20260928 16:44:45.955646 4143 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Bootstrap starting.
I20260928 16:44:45.956694 4142 tablet_bootstrap.cc:654] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:45.956890 4144 tablet_bootstrap.cc:654] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:45.957571 4142 log.cc:824] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:45.957711 4143 tablet_bootstrap.cc:654] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:45.957737 4144 log.cc:824] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:45.958441 4143 log.cc:824] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:45.959964 4144 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: No bootstrap required, opened a new log
I20260928 16:44:45.959964 4142 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: No bootstrap required, opened a new log
I20260928 16:44:45.960170 4144 ts_tablet_manager.cc:1397] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:44:45.960170 4142 ts_tablet_manager.cc:1397] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:44:45.960381 4143 tablet_bootstrap.cc:492] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: No bootstrap required, opened a new log
I20260928 16:44:45.960575 4143 ts_tablet_manager.cc:1397] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 16:44:45.964516 4143 raft_consensus.cc:348] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.964783 4143 raft_consensus.cc:374] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:45.964840 4143 raft_consensus.cc:729] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6be26657d4142cab8b1ac442aa9fec6, State: Initialized, Role: FOLLOWER
I20260928 16:44:45.964785 4142 raft_consensus.cc:348] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.964785 4144 raft_consensus.cc:348] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.965039 4142 raft_consensus.cc:374] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:45.965039 4144 raft_consensus.cc:374] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:45.965097 4144 raft_consensus.cc:729] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25145ac0f28b4d4d95289d8106d19124, State: Initialized, Role: FOLLOWER
I20260928 16:44:45.965106 4142 raft_consensus.cc:729] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf1e20fb0204f3bac1b5532d6649093, State: Initialized, Role: FOLLOWER
I20260928 16:44:45.965373 4143 consensus_queue.cc:260] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.965519 4142 consensus_queue.cc:260] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.965519 4144 consensus_queue.cc:260] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:45.966120 3994 heartbeater.cc:503] Master 127.30.196.62:46461 was elected leader, sending a full tablet report...
I20260928 16:44:45.966490 4144 ts_tablet_manager.cc:1428] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:44:45.966521 4143 ts_tablet_manager.cc:1428] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 16:44:45.966775 3863 heartbeater.cc:503] Master 127.30.196.62:46461 was elected leader, sending a full tablet report...
I20260928 16:44:45.966869 4125 heartbeater.cc:503] Master 127.30.196.62:46461 was elected leader, sending a full tablet report...
I20260928 16:44:45.968370 4142 ts_tablet_manager.cc:1428] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260928 16:44:45.971334 3995 tablet.cc:2369] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:46.031023 3864 tablet.cc:2369] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:46.031186 4149 raft_consensus.cc:482] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:46.031409 4149 raft_consensus.cc:504] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:46.032575 4149 leader_election.cc:288] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6be26657d4142cab8b1ac442aa9fec6 (127.30.196.2:37875), ecf1e20fb0204f3bac1b5532d6649093 (127.30.196.3:40395)
I20260928 16:44:46.034793 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.032930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37875 (local address 127.30.196.1:44179)
0928 16:44:46.033089 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:46.033108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:46.033207 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:46.033534 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:46.033537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:46.033553 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:46.034093 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:46.034095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:46.034104 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:46.034157 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:46.034496 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:46.034502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:46.034511 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:44:46.034604 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:44:46.034645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 16:44:46.034832 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.033183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:44179 (local address 127.30.196.2:37875)
0928 16:44:46.033322 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.033329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.033347 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:46.033386 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.033388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.033417 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.033519 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:46.034324 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:46.034328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:46.034331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:46.034345 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:46.034371 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:46.034383 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:46.034484 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:46.034623 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 16:44:46.034680 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:44:46.034978 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.032930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:40395 (local address 127.30.196.1:41943)
0928 16:44:46.033573 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:46.033590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:46.033701 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:46.033939 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:46.033942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:46.033956 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:46.034388 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:46.034390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:46.034401 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:46.034432 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:46.034703 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:46.034707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:46.034709 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 16:44:46.034786 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:44:46.034819 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":305,"thread_start_us":137,"threads_started":1}
I20260928 16:44:46.035046 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.033120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:41943 (local address 127.30.196.3:40395)
0928 16:44:46.033294 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.033300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.033676 (+ 376us) server_negotiation.cc:407] Connection header received
0928 16:44:46.033792 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.033795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.033830 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.033922 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:46.034560 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:46.034567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:46.034571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:46.034585 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:46.034605 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:46.034616 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:46.034698 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:46.034792 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 16:44:46.034837 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:46.035537 3948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fea68874d64e919043e1dae0f941de" candidate_uuid: "25145ac0f28b4d4d95289d8106d19124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6be26657d4142cab8b1ac442aa9fec6" is_pre_election: true
I20260928 16:44:46.035807 4079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fea68874d64e919043e1dae0f941de" candidate_uuid: "25145ac0f28b4d4d95289d8106d19124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf1e20fb0204f3bac1b5532d6649093" is_pre_election: true
I20260928 16:44:46.035913 3948 raft_consensus.cc:2454] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25145ac0f28b4d4d95289d8106d19124 in term 0.
I20260928 16:44:46.036075 4079 raft_consensus.cc:2454] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25145ac0f28b4d4d95289d8106d19124 in term 0.
I20260928 16:44:46.036475 3750 leader_election.cc:302] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25145ac0f28b4d4d95289d8106d19124, b6be26657d4142cab8b1ac442aa9fec6, ecf1e20fb0204f3bac1b5532d6649093; no voters:
I20260928 16:44:46.036744 4149 raft_consensus.cc:2781] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:46.036844 4149 raft_consensus.cc:482] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:46.036904 4149 raft_consensus.cc:3037] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:46.037990 4149 raft_consensus.cc:504] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:46.038381 4149 leader_election.cc:288] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [CANDIDATE]: Term 1 election: Requested vote from peers b6be26657d4142cab8b1ac442aa9fec6 (127.30.196.2:37875), ecf1e20fb0204f3bac1b5532d6649093 (127.30.196.3:40395)
I20260928 16:44:46.038751 3948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fea68874d64e919043e1dae0f941de" candidate_uuid: "25145ac0f28b4d4d95289d8106d19124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6be26657d4142cab8b1ac442aa9fec6"
I20260928 16:44:46.038753 4079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fea68874d64e919043e1dae0f941de" candidate_uuid: "25145ac0f28b4d4d95289d8106d19124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf1e20fb0204f3bac1b5532d6649093"
I20260928 16:44:46.038870 3948 raft_consensus.cc:3037] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:46.038899 4079 raft_consensus.cc:3037] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:46.040066 4079 raft_consensus.cc:2454] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25145ac0f28b4d4d95289d8106d19124 in term 1.
I20260928 16:44:46.040056 3948 raft_consensus.cc:2454] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25145ac0f28b4d4d95289d8106d19124 in term 1.
I20260928 16:44:46.040378 3752 leader_election.cc:302] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25145ac0f28b4d4d95289d8106d19124, ecf1e20fb0204f3bac1b5532d6649093; no voters:
I20260928 16:44:46.040588 4149 raft_consensus.cc:2781] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:46.040851 4149 raft_consensus.cc:686] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 1 LEADER]: Becoming Leader. State: Replica: 25145ac0f28b4d4d95289d8106d19124, State: Running, Role: LEADER
I20260928 16:44:46.041225 4149 consensus_queue.cc:237] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } }
I20260928 16:44:46.044550 3674 catalog_manager.cc:5495] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25145ac0f28b4d4d95289d8106d19124 (127.30.196.1). New cstate: current_term: 1 leader_uuid: "25145ac0f28b4d4d95289d8106d19124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25145ac0f28b4d4d95289d8106d19124" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:46.067390 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.065811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.30.196.1:36379)
0928 16:44:46.065974 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.065980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.066010 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:44:46.066106 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.066108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.066169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.066261 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:46.066958 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:46.066964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:46.066971 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:46.066983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:46.067001 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:46.067008 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:46.067114 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:46.067199 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 16:44:46.067240 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:44:46.073469 3948 raft_consensus.cc:1261] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 1 FOLLOWER]: Refusing update from remote peer 25145ac0f28b4d4d95289d8106d19124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:46.073469 4079 raft_consensus.cc:1261] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 1 FOLLOWER]: Refusing update from remote peer 25145ac0f28b4d4d95289d8106d19124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:46.074235 4152 consensus_queue.cc:1035] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf1e20fb0204f3bac1b5532d6649093" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:46.074437 4149 consensus_queue.cc:1035] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6be26657d4142cab8b1ac442aa9fec6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:46.085387 4156 mvcc.cc:205] Tried to move back new op lower bound from 7334354477349851136 to 7334354477227929600. Current Snapshot: MvccSnapshot[applied={T|T < 7334354477349851136 or (T in {7334354477349851136})}]
I20260928 16:44:46.092059 3674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:46.092278 3674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:46.095139 3674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b82ea84c3d84034bf8766a539f443dc: Sending DeleteTablet for 3 replicas of tablet 65fea68874d64e919043e1dae0f941de
I20260928 16:44:46.096062 3928 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fea68874d64e919043e1dae0f941de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:46 UTC) from {username='slave'} at 127.0.0.1:46434
I20260928 16:44:46.096197 4059 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fea68874d64e919043e1dae0f941de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:46 UTC) from {username='slave'} at 127.0.0.1:33458
I20260928 16:44:46.096678 4165 tablet_replica.cc:330] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: stopping tablet replica
I20260928 16:44:46.096683 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fea68874d64e919043e1dae0f941de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:46 UTC) from {username='slave'} at 127.0.0.1:37326
I20260928 16:44:46.097137 4167 tablet_replica.cc:330] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124: stopping tablet replica
I20260928 16:44:46.097281 4166 tablet_replica.cc:330] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: stopping tablet replica
I20260928 16:44:46.097419 4167 raft_consensus.cc:2229] T 65fea68874d64e919043e1dae0f941de P 25145ac0f28b4d4d95289d8106d19124 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:46.097596 4166 raft_consensus.cc:2229] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:46.097743 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3735
I20260928 16:44:46.098563 4166 raft_consensus.cc:2258] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:46.099614 4165 raft_consensus.cc:2229] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:46.099995 4165 raft_consensus.cc:2258] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:46.101176 4166 ts_tablet_manager.cc:1905] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:46.101248 4165 ts_tablet_manager.cc:1905] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:46.103544 4165 ts_tablet_manager.cc:1918] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:46.103706 4165 log.cc:1196] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/65fea68874d64e919043e1dae0f941de
I20260928 16:44:46.104069 4165 ts_tablet_manager.cc:1939] T 65fea68874d64e919043e1dae0f941de P b6be26657d4142cab8b1ac442aa9fec6: Deleting consensus metadata
I20260928 16:44:46.105476 4166 ts_tablet_manager.cc:1918] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:46.105487 3659 catalog_manager.cc:4841] TS b6be26657d4142cab8b1ac442aa9fec6 (127.30.196.2:37875): tablet 65fea68874d64e919043e1dae0f941de (table test-table [id=297652bd83824b4086937aac06b13f4e]) successfully deleted
I20260928 16:44:46.105625 4166 log.cc:1196] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/65fea68874d64e919043e1dae0f941de
I20260928 16:44:46.106014 4166 ts_tablet_manager.cc:1939] T 65fea68874d64e919043e1dae0f941de P ecf1e20fb0204f3bac1b5532d6649093: Deleting consensus metadata
W20260928 16:44:46.106696 3661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:36379 (error 108)
I20260928 16:44:46.107380 3661 catalog_manager.cc:4841] TS ecf1e20fb0204f3bac1b5532d6649093 (127.30.196.3:40395): tablet 65fea68874d64e919043e1dae0f941de (table test-table [id=297652bd83824b4086937aac06b13f4e]) successfully deleted
I20260928 16:44:46.107647 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3866
I20260928 16:44:46.108585 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.107800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:36379 (local address 127.0.0.1:37352)
0928 16:44:46.108369 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:46.108469 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
W20260928 16:44:46.108810 3661 catalog_manager.cc:4584] TS 25145ac0f28b4d4d95289d8106d19124 (127.30.196.1:36379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65fea68874d64e919043e1dae0f941de: Network error: Client connection negotiation failed: client connection to 127.30.196.1:36379: connect: Connection refused (error 111)
I20260928 16:44:46.114157 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3997
I20260928 16:44:46.121177 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 3644
2026-09-28T16:44:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1037 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 16:44:46.143630 31504 mini_oidc.cc:152] Starting JWKS server
I20260928 16:44:46.145363 31504 webserver.cc:460] Webserver started at https://127.0.0.1:41639/ using document root <none> and password file <none>
I20260928 16:44:46.145560 31504 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:44:46.145879 31504 webserver.cc:460] Webserver started at http://127.0.0.1:32853/ using document root <none> and password file <none>
2026-09-28T16:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:46Z Disabled control of system clock
I20260928 16:44:46.155577 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45421
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:45421
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41639/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/testchainca.cert with env {}
W20260928 16:44:46.270223 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:46.270562 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:46.270609 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:46.275034 4174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:46.275105 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:46.275120 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:46.275137 4174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:46.275153 4174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:44:46.275170 4174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:44:46.275189 4174 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41639/jwks.json
I20260928 16:44:46.280699 4174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:45421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41639/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:46.282008 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:46.283110 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:46.292227 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:46.292232 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:46.292238 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:46.292773 4174 server_base.cc:1034] running on GCE node
I20260928 16:44:46.293387 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:46.294570 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:46.295749 4174 hybrid_clock.cc:648] HybridClock initialized: now 1790613886295725 us; error 34 us; skew 500 ppm
I20260928 16:44:46.297739 4174 webserver.cc:460] Webserver started at http://127.30.196.62:43349/ using document root <none> and password file <none>
I20260928 16:44:46.298398 4174 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:46.298512 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:46.298743 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:46.301431 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "f03c0559b412bb5f294328a071dc0359"
server_key_iv: "c4a3e6f52934b6f1982f851adc1799c3"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.302063 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "f03c0559b412bb5f294328a071dc0359"
server_key_iv: "c4a3e6f52934b6f1982f851adc1799c3"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.306098 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:44:46.308740 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:46.310129 4174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:46.310302 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "f03c0559b412bb5f294328a071dc0359"
server_key_iv: "c4a3e6f52934b6f1982f851adc1799c3"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.310436 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:46.363374 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:46.364179 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:46.364390 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:46.371786 4174 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:45421
I20260928 16:44:46.371846 4240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:45421 every 8 connection(s)
I20260928 16:44:46.372845 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:46.376092 4241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:46.381850 4241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051: Bootstrap starting.
I20260928 16:44:46.382810 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4174
I20260928 16:44:46.382942 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:46.383286 31504 external_mini_cluster.cc:1428] Setting key da162f739e3891750369028a5bf62973
I20260928 16:44:46.385010 4241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:46.385821 4241 log.cc:824] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:46.388013 4241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051: No bootstrap required, opened a new log
I20260928 16:44:46.392477 4241 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } }
I20260928 16:44:46.392719 4241 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:46.392781 4241 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaa7d8a5d2b4dbfa360a1530a13c051, State: Initialized, Role: FOLLOWER
I20260928 16:44:46.392910 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.384451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.30.196.62:45421)
0928 16:44:46.385190 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.385205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.385246 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:44:46.385926 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.385953 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.386448 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.386799 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:46.387146 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.388231 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.389454 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.389924 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.390043 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.390929 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:46.390934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:46.390947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:46.391338 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:46.391398 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:46.391418 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:46.391500 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:46.391753 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 16:44:46.392063 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260928 16:44:46.393362 4241 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } }
I20260928 16:44:46.393544 4241 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:46.393664 4241 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:46.393793 4241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:46.394824 4241 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } }
I20260928 16:44:46.395241 4241 leader_election.cc:302] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbaa7d8a5d2b4dbfa360a1530a13c051; no voters:
I20260928 16:44:46.395619 4241 leader_election.cc:288] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:46.395758 4246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:46.396080 4246 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [term 1 LEADER]: Becoming Leader. State: Replica: fbaa7d8a5d2b4dbfa360a1530a13c051, State: Running, Role: LEADER
I20260928 16:44:46.396462 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:46.396416 4246 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } }
I20260928 16:44:46.397028 4241 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:46.398473 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbaa7d8a5d2b4dbfa360a1530a13c051. Latest consensus state: current_term: 1 leader_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } } }
I20260928 16:44:46.398500 4247 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa7d8a5d2b4dbfa360a1530a13c051" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45421 } } }
I20260928 16:44:46.398645 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:46.398684 4247 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:46.399003 4255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:46.402279 4255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:46.408349 4255 catalog_manager.cc:1347] Generated new cluster ID: d9893ec2cb1040b694ce7cdb8ed93501
I20260928 16:44:46.408445 4255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:46.433802 4255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:46.434885 4255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:46.455720 4255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbaa7d8a5d2b4dbfa360a1530a13c051: Generated new TSK 0
I20260928 16:44:46.456600 4255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:44:46.490968 4271 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 16:44:46.491110 4271 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:44:46.491719 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.474211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.30.196.62:45421)
0928 16:44:46.474362 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.474369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.474599 (+ 230us) server_negotiation.cc:407] Connection header received
0928 16:44:46.474820 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.474823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.474869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.474964 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:44:46.475274 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.475873 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.477273 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.477495 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.477583 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.477997 (+ 414us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:44:46.491257 (+ 13260us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:44:46.491544 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:44:46.491824 4244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 16:44:46.493603 4270 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:46.474122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45421 (local address 127.0.0.1:33540)
0928 16:44:46.474504 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:46.474521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:46.474663 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:46.475010 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:46.475013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:46.475022 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:44:46.475163 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:46.475170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.476008 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.476013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:46.477124 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:46.477136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.477631 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.477636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:46.477742 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.477799 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:44:46.491549 (+ 13750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:44:46.491848 (+ 299us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.196.62:45421: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.196.62:45421: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41639/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 16:44:46.494556 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4174
2026-09-28T16:44:46Z chronyd exiting
I20260928 16:44:46.532727 31504 test_util.cc:175] -----------------------------------------------
I20260928 16:44:46.532832 31504 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613859608463-31504-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (390 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 16:44:46.533913 31504 mini_oidc.cc:152] Starting JWKS server
I20260928 16:44:46.534792 31504 webserver.cc:460] Webserver started at https://127.0.0.1:44209/ using document root <none> and password file <none>
I20260928 16:44:46.534922 31504 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:44:46.535193 31504 webserver.cc:460] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
2026-09-28T16:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:46Z Disabled control of system clock
I20260928 16:44:46.544413 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34967
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34967
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/testchainca.cert with env {}
W20260928 16:44:46.658207 4277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:46.658533 4277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:46.658581 4277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:46.662904 4277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:46.662983 4277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:46.662999 4277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:46.663017 4277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:46.663033 4277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:44:46.663048 4277 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:44:46.663122 4277 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44209/jwks.json
I20260928 16:44:46.668859 4277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:46.670043 4277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:46.671161 4277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:46.680161 4282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:46.680195 4283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:46.680299 4277 server_base.cc:1034] running on GCE node
W20260928 16:44:46.680213 4285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:46.681106 4277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:46.682348 4277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:46.683563 4277 hybrid_clock.cc:648] HybridClock initialized: now 1790613886683552 us; error 45 us; skew 500 ppm
I20260928 16:44:46.685688 4277 webserver.cc:460] Webserver started at http://127.30.196.62:46119/ using document root <none> and password file <none>
I20260928 16:44:46.686400 4277 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:46.686535 4277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:46.686777 4277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:46.688982 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "c2714bcc812a4b3cb753b6d00c937764"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "af148986b4e51927ba29479894a2ead2"
server_key_iv: "f7a006402b8f42463f61e00e07b7bc97"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.689570 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "c2714bcc812a4b3cb753b6d00c937764"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "af148986b4e51927ba29479894a2ead2"
server_key_iv: "f7a006402b8f42463f61e00e07b7bc97"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.693437 4277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:46.696091 4291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:46.697383 4277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:46.697526 4277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "c2714bcc812a4b3cb753b6d00c937764"
format_stamp: "Formatted at 2026-09-28 16:44:46 on dist-test-slave-5rg1"
server_key: "af148986b4e51927ba29479894a2ead2"
server_key_iv: "f7a006402b8f42463f61e00e07b7bc97"
server_key_version: "encryptionkey@0"
I20260928 16:44:46.697685 4277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:46.723285 4277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:46.724154 4277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:46.724367 4277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:46.731647 4277 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34967
I20260928 16:44:46.731701 4343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34967 every 8 connection(s)
I20260928 16:44:46.732762 4277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:46.736446 4344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:46.741127 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4277
I20260928 16:44:46.741243 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:46.741554 31504 external_mini_cluster.cc:1428] Setting key 853ea3ac9ecf330d90036db2be88c0f8
I20260928 16:44:46.742933 4344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764: Bootstrap starting.
I20260928 16:44:46.745302 4344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:46.746084 4344 log.cc:824] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:46.748078 4344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764: No bootstrap required, opened a new log
I20260928 16:44:46.751026 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.742976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.30.196.62:34967)
0928 16:44:46.743725 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.743751 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.743789 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:44:46.744451 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.744477 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.744884 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.745241 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:46.745689 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.746739 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.747640 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.748208 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.748401 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.749303 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:46.749316 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:46.749329 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:46.749735 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:46.749787 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:46.749807 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:46.749920 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:46.750051 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:44:46.750306 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260928 16:44:46.751786 4344 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } }
I20260928 16:44:46.752039 4344 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:46.752091 4344 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2714bcc812a4b3cb753b6d00c937764, State: Initialized, Role: FOLLOWER
I20260928 16:44:46.752545 4344 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } }
I20260928 16:44:46.752705 4344 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:46.752769 4344 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:46.752894 4344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:46.754124 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:46.754165 4344 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } }
I20260928 16:44:46.754534 4344 leader_election.cc:302] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2714bcc812a4b3cb753b6d00c937764; no voters:
I20260928 16:44:46.754887 4344 leader_election.cc:288] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:46.754967 4349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:46.755216 4349 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [term 1 LEADER]: Becoming Leader. State: Replica: c2714bcc812a4b3cb753b6d00c937764, State: Running, Role: LEADER
I20260928 16:44:46.755541 4349 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } }
I20260928 16:44:46.756330 4344 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:46.756784 4349 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2714bcc812a4b3cb753b6d00c937764. Latest consensus state: current_term: 1 leader_uuid: "c2714bcc812a4b3cb753b6d00c937764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } } }
I20260928 16:44:46.756906 4349 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:46.757220 4350 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2714bcc812a4b3cb753b6d00c937764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2714bcc812a4b3cb753b6d00c937764" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34967 } } }
I20260928 16:44:46.757297 4353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:46.757340 4350 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:46.760627 4353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:46.766319 4353 catalog_manager.cc:1347] Generated new cluster ID: ad5f2f29d8c4479c870128208f3140df
I20260928 16:44:46.766417 4353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:46.775327 4353 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:46.776588 4353 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:46.789574 4353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2714bcc812a4b3cb753b6d00c937764: Generated new TSK 0
I20260928 16:44:46.790534 4353 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:44:46.818485 4374 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 16:44:46.818635 4374 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:44:46.819695 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:46.802861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.30.196.62:34967)
0928 16:44:46.803004 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:46.803009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:46.803178 (+ 169us) server_negotiation.cc:407] Connection header received
0928 16:44:46.803319 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:46.803322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:46.803387 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:46.803522 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:44:46.803802 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.804642 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.805547 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.805896 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.806002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.806308 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:44:46.819146 (+ 12838us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:44:46.819500 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41836: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:44:46.819773 4347 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41836: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 16:44:46.819960 4373 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:46.802744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34967 (local address 127.0.0.1:41836)
0928 16:44:46.803057 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:46.803070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:46.803206 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:46.803525 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:46.803527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:46.803534 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:44:46.803640 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:46.803645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.804775 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.804778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:46.805431 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:46.805438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:46.806028 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:46.806031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:46.806099 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:46.806124 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:44:46.819398 (+ 13274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:44:46.819720 (+ 322us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.196.62:34967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260928 16:44:46.820716 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4277
2026-09-28T16:44:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 16:44:46.861725 31504 mini_oidc.cc:152] Starting JWKS server
I20260928 16:44:46.862669 31504 webserver.cc:460] Webserver started at https://127.0.0.1:39741/ using document root <none> and password file <none>
I20260928 16:44:46.862789 31504 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:44:46.863077 31504 webserver.cc:460] Webserver started at http://127.0.0.1:46183/ using document root <none> and password file <none>
2026-09-28T16:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:46Z Disabled control of system clock
I20260928 16:44:46.872789 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34419
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34419
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39741/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:44:46.986922 4380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:46.987260 4380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:46.987331 4380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:46.991456 4380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:46.991562 4380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:46.991595 4380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:46.991634 4380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:46.991689 4380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:44:46.991729 4380 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:44:46.991756 4380 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39741/jwks.json
I20260928 16:44:46.996773 4380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39741/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:46.997942 4380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:46.999442 4380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:47.008383 4385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.008402 4388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.008383 4386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:47.008828 4380 server_base.cc:1034] running on GCE node
I20260928 16:44:47.009406 4380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:47.010618 4380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:47.011751 4380 hybrid_clock.cc:648] HybridClock initialized: now 1790613887011730 us; error 41 us; skew 500 ppm
I20260928 16:44:47.013634 4380 webserver.cc:460] Webserver started at http://127.30.196.62:36465/ using document root <none> and password file <none>
I20260928 16:44:47.014201 4380 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:47.014269 4380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:47.014436 4380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:47.016559 4380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "9829895893e61403e50d6f16ff75e1eb"
server_key_iv: "8a06980a45c5ce83160c659419919a5e"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.017087 4380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "9829895893e61403e50d6f16ff75e1eb"
server_key_iv: "8a06980a45c5ce83160c659419919a5e"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.021061 4380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:47.023527 4394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:47.024926 4380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:47.025038 4380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "9829895893e61403e50d6f16ff75e1eb"
server_key_iv: "8a06980a45c5ce83160c659419919a5e"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.025147 4380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:47.059607 4380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:47.060442 4380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:47.060658 4380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:47.068246 4380 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34419
I20260928 16:44:47.068290 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34419 every 8 connection(s)
I20260928 16:44:47.069274 4380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:47.072700 4447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:47.079118 4447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5: Bootstrap starting.
I20260928 16:44:47.079437 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4380
I20260928 16:44:47.079604 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:47.079883 31504 external_mini_cluster.cc:1428] Setting key b203a372b9cc3e29cf27453cd55fcbc1
I20260928 16:44:47.082077 4447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:47.082911 4447 log.cc:824] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:47.085088 4447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5: No bootstrap required, opened a new log
I20260928 16:44:47.089974 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:47.081250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.30.196.62:34419)
0928 16:44:47.081931 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:47.081946 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:47.081980 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:44:47.082694 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:47.082712 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:47.083160 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:47.083585 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:47.084012 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.085031 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.086380 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.086879 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.087032 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.087945 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:47.087951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:47.087964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:47.088363 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:47.088419 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:47.088440 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:47.088528 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:47.088714 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:44:47.089048 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":163,"threads_started":1}
I20260928 16:44:47.090255 4447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } }
I20260928 16:44:47.090519 4447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:47.090610 4447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de5fa6b6eb64d20b6d49ee69c5c62f5, State: Initialized, Role: FOLLOWER
I20260928 16:44:47.091106 4447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } }
I20260928 16:44:47.091284 4447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:47.091384 4447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:47.091511 4447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:47.092756 4447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } }
I20260928 16:44:47.093145 4447 leader_election.cc:302] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de5fa6b6eb64d20b6d49ee69c5c62f5; no voters:
I20260928 16:44:47.093240 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:47.093582 4447 leader_election.cc:288] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:47.093716 4452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:47.093976 4452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [term 1 LEADER]: Becoming Leader. State: Replica: 8de5fa6b6eb64d20b6d49ee69c5c62f5, State: Running, Role: LEADER
I20260928 16:44:47.094338 4452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } }
I20260928 16:44:47.094977 4447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:47.096436 4454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de5fa6b6eb64d20b6d49ee69c5c62f5. Latest consensus state: current_term: 1 leader_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } } }
I20260928 16:44:47.096403 4453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de5fa6b6eb64d20b6d49ee69c5c62f5" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34419 } } }
I20260928 16:44:47.096587 4454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.096596 4453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.097049 4461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:47.100553 4461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:47.106220 4461 catalog_manager.cc:1347] Generated new cluster ID: 8c87241f9f9e4ad88bcd46a83e796abc
I20260928 16:44:47.106310 4461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:47.120289 4461 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:47.121459 4461 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:47.126494 4461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de5fa6b6eb64d20b6d49ee69c5c62f5: Generated new TSK 0
I20260928 16:44:47.127276 4461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:47.146162 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:47.140864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.30.196.62:34419)
0928 16:44:47.141015 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:47.141023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:47.141161 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:44:47.141346 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:47.141349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:47.141404 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:47.141504 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:44:47.141815 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.142351 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.143763 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.143987 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.144071 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.144420 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:44:47.145724 (+ 1304us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39741/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:44:47.145988 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39741/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 16:44:47.146238 4476 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:47.140691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34419 (local address 127.0.0.1:38238)
0928 16:44:47.141056 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:47.141071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:47.141197 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:47.141553 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:47.141557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:47.141565 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:44:47.141694 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:47.141700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.142525 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.142529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:47.143592 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:47.143603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.144113 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.144117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:47.144211 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.144242 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:44:47.145897 (+ 1655us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39741/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:44:47.146050 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.196.62:34419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39741/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
W20260928 16:44:47.146250 4450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39741/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 16:44:47.146908 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4380
E20260928 16:44:47.157874 4477 webserver.cc:533] Webserver: sslize failed:
E20260928 16:44:47.157975 4477 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T16:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T16:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:47Z Disabled control of system clock
I20260928 16:44:47.195083 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38137
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38137
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/testchainca.cert with env {}
W20260928 16:44:47.312755 4481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:47.313130 4481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:47.313225 4481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:47.317674 4481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:47.317787 4481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:47.317816 4481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:47.317858 4481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:47.317898 4481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:44:47.317958 4481 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:44:47.317991 4481 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 16:44:47.325496 4481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:47.326738 4481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:47.327893 4481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:47.337330 4487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.337328 4489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.337328 4486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:47.337827 4481 server_base.cc:1034] running on GCE node
I20260928 16:44:47.338464 4481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:47.339643 4481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:47.340893 4481 hybrid_clock.cc:648] HybridClock initialized: now 1790613887340859 us; error 52 us; skew 500 ppm
I20260928 16:44:47.343087 4481 webserver.cc:460] Webserver started at http://127.30.196.62:42781/ using document root <none> and password file <none>
I20260928 16:44:47.343809 4481 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:47.343912 4481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:47.344159 4481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:47.346299 4481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "3e6ff8bb3c9642219563a95e65f9adcd"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "7a87a3ce9a396858556b7ba8e26bd375"
server_key_iv: "63c865683163dce6515b5df55ed3e15c"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.346931 4481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "3e6ff8bb3c9642219563a95e65f9adcd"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "7a87a3ce9a396858556b7ba8e26bd375"
server_key_iv: "63c865683163dce6515b5df55ed3e15c"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.350998 4481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 16:44:47.353628 4495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:47.355053 4481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:47.355237 4481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "3e6ff8bb3c9642219563a95e65f9adcd"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-5rg1"
server_key: "7a87a3ce9a396858556b7ba8e26bd375"
server_key_iv: "63c865683163dce6515b5df55ed3e15c"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.355371 4481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:47.377368 4481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:47.378196 4481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:47.378427 4481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:47.386098 4481 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38137
I20260928 16:44:47.386149 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38137 every 8 connection(s)
I20260928 16:44:47.387120 4481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:47.390372 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:47.391543 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4481
I20260928 16:44:47.391718 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:47.392026 31504 external_mini_cluster.cc:1428] Setting key 50ad89e4b01342727f415182c841f95f
I20260928 16:44:47.397234 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd: Bootstrap starting.
I20260928 16:44:47.399879 4548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:47.400719 4548 log.cc:824] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:47.401911 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:47.393555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.30.196.62:38137)
0928 16:44:47.394277 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:47.394294 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:47.394330 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:44:47.394976 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:47.395005 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:47.395413 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:47.395843 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:47.396307 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.397287 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.398226 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.398705 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.398892 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.399858 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:47.399863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:47.399877 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:47.400350 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:47.400413 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:47.400434 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:47.400553 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:47.400697 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 16:44:47.400995 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":155,"threads_started":1}
I20260928 16:44:47.403043 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd: No bootstrap required, opened a new log
I20260928 16:44:47.405114 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:44:47.407642 4548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } }
I20260928 16:44:47.407917 4548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:47.408004 4548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6ff8bb3c9642219563a95e65f9adcd, State: Initialized, Role: FOLLOWER
I20260928 16:44:47.408547 4548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } }
I20260928 16:44:47.408723 4548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:47.408826 4548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:47.408977 4548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:47.410048 4548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } }
I20260928 16:44:47.410436 4548 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6ff8bb3c9642219563a95e65f9adcd; no voters:
I20260928 16:44:47.410799 4548 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:47.410960 4553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:47.411275 4553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [term 1 LEADER]: Becoming Leader. State: Replica: 3e6ff8bb3c9642219563a95e65f9adcd, State: Running, Role: LEADER
I20260928 16:44:47.411600 4553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } }
I20260928 16:44:47.412315 4548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:47.413197 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6ff8bb3c9642219563a95e65f9adcd. Latest consensus state: current_term: 1 leader_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } } }
I20260928 16:44:47.413352 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.413720 4554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6ff8bb3c9642219563a95e65f9adcd" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38137 } } }
I20260928 16:44:47.413832 4554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.414084 4562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:47.417936 4562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:47.424281 4562 catalog_manager.cc:1347] Generated new cluster ID: c794901d86d240fd8934136374b0a22b
I20260928 16:44:47.424398 4562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:47.444382 4562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:47.445590 4562 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:47.452862 4562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e6ff8bb3c9642219563a95e65f9adcd: Generated new TSK 0
I20260928 16:44:47.453680 4562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:47.493283 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:47.467918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.30.196.62:38137)
0928 16:44:47.468120 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:47.468125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:47.468327 (+ 202us) server_negotiation.cc:407] Connection header received
0928 16:44:47.468522 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:47.468525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:47.468577 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:47.468669 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:44:47.468952 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.469579 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.470903 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.471205 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.471304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.471766 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:44:47.492806 (+ 21040us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:44:47.493160 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":102}
W20260928 16:44:47.493399 4551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 16:44:47.493899 4577 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:47.467842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38137 (local address 127.0.0.1:33628)
0928 16:44:47.468223 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:47.468239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:47.468366 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:47.468696 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:47.468699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:47.468708 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:44:47.468827 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:47.468833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.469702 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.469706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:47.470747 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:47.470758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.471355 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.471359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:47.471462 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.471500 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:44:47.493123 (+ 21623us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:44:47.493352 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.196.62:38137: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":185,"threads_started":1}
I20260928 16:44:47.494535 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4481
2026-09-28T16:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (327 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:47 dist-test-slave-5rg1 krb5kdc[4589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:47 dist-test-slave-5rg1 krb5kdc[4589](info): set up 2 sockets
Sep 28 16:44:47 dist-test-slave-5rg1 krb5kdc[4589](info): commencing operation
krb5kdc: starting...
W20260928 16:44:48.804335 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:48 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:44:48.935326 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34303
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:49.046289 4602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:49.046581 4602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:49.046623 4602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:49.050699 4602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:49.050763 4602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:49.050782 4602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:49.050798 4602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:49.050814 4602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:49.056273 4602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:49.057426 4602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:49.058473 4602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:49.067119 4608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.067102 4602 server_base.cc:1034] running on GCE node
W20260928 16:44:49.067028 4607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.067024 4610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.068025 4602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:49.069197 4602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:49.070385 4602 hybrid_clock.cc:648] HybridClock initialized: now 1790613889070361 us; error 46 us; skew 500 ppm
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.073278 4602 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:44:49.074230 4602 webserver.cc:460] Webserver started at http://127.30.196.62:45363/ using document root <none> and password file <none>
I20260928 16:44:49.074808 4602 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:49.074901 4602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:49.075146 4602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:49.077527 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "e8f90f5bb4f249fa9c2be61f29410455"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "f13d83feedd8fb0beb3c52cdeeb725f7"
server_key_iv: "5fccc0c939a21ae51a0f9ab5c15d600f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.078179 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "e8f90f5bb4f249fa9c2be61f29410455"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "f13d83feedd8fb0beb3c52cdeeb725f7"
server_key_iv: "5fccc0c939a21ae51a0f9ab5c15d600f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.082115 4602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:49.084693 4617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.086014 4602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:49.086180 4602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "e8f90f5bb4f249fa9c2be61f29410455"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "f13d83feedd8fb0beb3c52cdeeb725f7"
server_key_iv: "5fccc0c939a21ae51a0f9ab5c15d600f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.086305 4602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:49.110693 4602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:49.114182 4602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:49.114410 4602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:49.122035 4602 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34303
I20260928 16:44:49.122051 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34303 every 8 connection(s)
I20260928 16:44:49.123072 4602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:49.126221 4670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:49.132346 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4602
I20260928 16:44:49.132387 4670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455: Bootstrap starting.
I20260928 16:44:49.132503 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:49.132805 31504 external_mini_cluster.cc:1428] Setting key db17a9d4c7f2d121c11678e7c49d0fdd
I20260928 16:44:49.135299 4670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:49.136200 4670 log.cc:824] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:49.138242 4670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455: No bootstrap required, opened a new log
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:49.143217 4670 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:44:49.143533 4670 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:49.143632 4670 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f90f5bb4f249fa9c2be61f29410455, State: Initialized, Role: FOLLOWER
I20260928 16:44:49.144281 4670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:44:49.144477 4670 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:49.144555 4670 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:49.144685 4670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:49.145810 4670 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:44:49.146238 4670 leader_election.cc:302] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f90f5bb4f249fa9c2be61f29410455; no voters:
I20260928 16:44:49.146679 4670 leader_election.cc:288] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:49.146792 4675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:49.147060 4675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [term 1 LEADER]: Becoming Leader. State: Replica: e8f90f5bb4f249fa9c2be61f29410455, State: Running, Role: LEADER
I20260928 16:44:49.147464 4675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:44:49.148301 4670 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:49.149398 4677 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f90f5bb4f249fa9c2be61f29410455. Latest consensus state: current_term: 1 leader_uuid: "e8f90f5bb4f249fa9c2be61f29410455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } } }
I20260928 16:44:49.149544 4677 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:49.149398 4676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f90f5bb4f249fa9c2be61f29410455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f90f5bb4f249fa9c2be61f29410455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } } }
I20260928 16:44:49.149821 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.134320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.30.196.62:34303)
0928 16:44:49.134860 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.134881 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.134922 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:44:49.135815 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.135837 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.136216 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.136650 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.137577 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.138624 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.139491 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.140055 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.140236 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.143382 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.143388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.143404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.143437 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.145496 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.146003 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.146009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.146014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.146144 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.146477 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.146481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.146483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.147164 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.148089 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.148260 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.148646 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":164,"threads_started":1}
I20260928 16:44:49.149873 4676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:49.149930 4683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:49.153136 4683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:49.161533 4683 catalog_manager.cc:1347] Generated new cluster ID: 6e98e8ca7a0c46b685ee6907a469d435
I20260928 16:44:49.161633 4683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:49.182094 4683 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:49.183425 4683 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:49.190191 4683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f90f5bb4f249fa9c2be61f29410455: Generated new TSK 0
I20260928 16:44:49.191032 4683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:44:49.235230 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:49.353029 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:49.353310 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:49.353355 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:49.357620 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:49.357688 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:49.357784 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:49.363349 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:49.364538 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:49.365588 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:49.375931 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.376030 4696 server_base.cc:1034] running on GCE node
W20260928 16:44:49.375969 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.375973 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.376603 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:49.377250 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:49.378427 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790613889378396 us; error 47 us; skew 500 ppm
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.381527 4696 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:44:49.382524 4696 webserver.cc:460] Webserver started at http://127.30.196.1:32841/ using document root <none> and password file <none>
I20260928 16:44:49.383077 4696 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:49.383136 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:49.383302 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:49.385442 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "9a077488516841058fa86bd663029d1a"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "03e58fa37fe1a2e13ad9b114e3c8ce29"
server_key_iv: "0cd1db4518fa292e78d9126890b4646f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.385960 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "9a077488516841058fa86bd663029d1a"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "03e58fa37fe1a2e13ad9b114e3c8ce29"
server_key_iv: "0cd1db4518fa292e78d9126890b4646f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.389807 4696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 16:44:49.392545 4711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.393908 4696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:49.394105 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "9a077488516841058fa86bd663029d1a"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "03e58fa37fe1a2e13ad9b114e3c8ce29"
server_key_iv: "0cd1db4518fa292e78d9126890b4646f"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.394233 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:49.407341 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:49.410902 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:49.411144 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:49.411815 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:49.412722 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:49.412796 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.412870 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:49.412925 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.422434 4696 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:40547
I20260928 16:44:49.422474 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:40547 every 8 connection(s)
I20260928 16:44:49.423405 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:49.432535 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4696
I20260928 16:44:49.432680 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:49.432994 31504 external_mini_cluster.cc:1428] Setting key 29cfa58955cb88cb10f39b3ec9e2e403
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:49.438251 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.425509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:51391 (local address 127.30.196.62:34303)
0928 16:44:49.425657 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.425661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.426568 (+ 907us) server_negotiation.cc:407] Connection header received
0928 16:44:49.428123 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.428127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.428193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.428273 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.429517 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.430071 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.431264 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.431513 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.431647 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.434603 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.434606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.434608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.434631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.436152 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.436687 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.436692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.436694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.436775 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.437137 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.437141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.437143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.437327 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.437439 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.437941 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.438081 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:49.439090 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.425941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.1:51391)
0928 16:44:49.426413 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.426459 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.427894 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.428451 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.428458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.428837 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:49.429350 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.429364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.430211 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.430215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.431118 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.431128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.431724 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.431729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.431900 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.432606 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:49.432636 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:49.434416 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:49.436281 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.436287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.436303 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.436578 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.436877 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.436880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.436883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.437036 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.437437 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:49.437445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:49.437706 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.437901 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.438244 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260928 16:44:49.440452 4825 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
I20260928 16:44:49.440738 4825 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:49.441411 4825 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:49.443322 4634 ts_manager.cc:194] Registered new tserver with Master: 9a077488516841058fa86bd663029d1a (127.30.196.1:40547)
I20260928 16:44:49.444589 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:51391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:44:49.463740 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:49.579408 4830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:49.579793 4830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:49.579845 4830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:49.583962 4830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:49.584054 4830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:49.584152 4830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:49.589720 4830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:49.590948 4830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:49.592114 4830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:49.602288 4838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.602262 4836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.602246 4835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.602545 4830 server_base.cc:1034] running on GCE node
I20260928 16:44:49.602905 4830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:49.603539 4830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:49.604700 4830 hybrid_clock.cc:648] HybridClock initialized: now 1790613889604669 us; error 42 us; skew 500 ppm
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.613417 4830 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:44:49.614591 4830 webserver.cc:460] Webserver started at http://127.30.196.2:40137/ using document root <none> and password file <none>
I20260928 16:44:49.615197 4830 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:49.615262 4830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:49.615499 4830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:49.617784 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "1dbfa9312e904ca3bc3eff456ee634fb"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "dd07106bb3b46ad1bb371d72d530d813"
server_key_iv: "d9981ee767f528952ced5cd86534d3ce"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.618371 4830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "1dbfa9312e904ca3bc3eff456ee634fb"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "dd07106bb3b46ad1bb371d72d530d813"
server_key_iv: "d9981ee767f528952ced5cd86534d3ce"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.622510 4830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 16:44:49.625286 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.626746 4830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:49.626886 4830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "1dbfa9312e904ca3bc3eff456ee634fb"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "dd07106bb3b46ad1bb371d72d530d813"
server_key_iv: "d9981ee767f528952ced5cd86534d3ce"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.627017 4830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:49.638476 4830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:49.641721 4830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:49.641932 4830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:49.642570 4830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:49.643414 4830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:49.643462 4830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.643556 4830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:49.643594 4830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.653391 4830 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:36251
I20260928 16:44:49.653410 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:36251 every 8 connection(s)
I20260928 16:44:49.654342 4830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:49.660962 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4830
I20260928 16:44:49.661128 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:49.661449 31504 external_mini_cluster.cc:1428] Setting key f72d3a41999e40fb911d3758ff1af239
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:49.669417 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.656238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:52995 (local address 127.30.196.62:34303)
0928 16:44:49.656398 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.656403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.657382 (+ 979us) server_negotiation.cc:407] Connection header received
0928 16:44:49.658744 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.658747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.658797 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.658878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.660193 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.660715 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.662322 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.662620 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.662734 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.665811 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.665814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.665816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.665842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.667243 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.667823 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.667827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.667828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.667909 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.668271 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.668274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.668276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.668464 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.668581 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.669119 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.669248 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:44:49.670241 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.656643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.2:52995)
0928 16:44:49.657215 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.657258 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.658531 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.659065 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.659072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.659447 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:49.660020 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.660034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.660852 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.660856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.662191 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.662204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.662806 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.662813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.663015 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.663762 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:49.663788 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:49.665620 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:49.667367 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.667373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.667390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.667714 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.668011 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.668014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.668017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.668167 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.668580 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:49.668588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:49.668848 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.669060 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.669402 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":114,"threads_started":1}
I20260928 16:44:49.671418 4959 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:44:49.671727 4959 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:49.672299 4959 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
I20260928 16:44:49.673444 4634 ts_manager.cc:194] Registered new tserver with Master: 1dbfa9312e904ca3bc3eff456ee634fb (127.30.196.2:36251)
I20260928 16:44:49.674176 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:52995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:44:49.690898 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:49.806368 4964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:49.806757 4964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:49.806807 4964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:49.810968 4964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:49.811046 4964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:49.811116 4964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:49.817060 4964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:49.818264 4964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:49.819334 4964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:49.829662 4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.829595 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.829600 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.829782 4964 server_base.cc:1034] running on GCE node
I20260928 16:44:49.830502 4964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:49.831310 4964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:49.832559 4964 hybrid_clock.cc:648] HybridClock initialized: now 1790613889832512 us; error 58 us; skew 500 ppm
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.835537 4964 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:44:49.836551 4964 webserver.cc:460] Webserver started at http://127.30.196.3:33873/ using document root <none> and password file <none>
I20260928 16:44:49.837203 4964 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:49.837266 4964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:49.837510 4964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:49.839643 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "474517e84526471395fbe63ddd2f46f0"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "67e96594fd227e89544501757aa9d1da"
server_key_iv: "68ed77529c7d6c6e989e5174e1c5c200"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.840298 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "474517e84526471395fbe63ddd2f46f0"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "67e96594fd227e89544501757aa9d1da"
server_key_iv: "68ed77529c7d6c6e989e5174e1c5c200"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.844437 4964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:49.847173 4979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.848664 4964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:49.848814 4964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "474517e84526471395fbe63ddd2f46f0"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-5rg1"
server_key: "67e96594fd227e89544501757aa9d1da"
server_key_iv: "68ed77529c7d6c6e989e5174e1c5c200"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.848963 4964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:49.866192 4964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:49.869899 4964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:49.870128 4964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:49.870786 4964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:49.871747 4964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:49.871797 4964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.871858 4964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:49.871889 4964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.881431 4964 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:34695
I20260928 16:44:49.881448 5092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:34695 every 8 connection(s)
I20260928 16:44:49.882459 4964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:49.888164 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 4964
I20260928 16:44:49.888330 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:49.888621 31504 external_mini_cluster.cc:1428] Setting key 4dc34fbed70854a37e6f2b5f5083fbf0
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:44:49.897614 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.884816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:39719 (local address 127.30.196.62:34303)
0928 16:44:49.884961 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.884964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.885909 (+ 945us) server_negotiation.cc:407] Connection header received
0928 16:44:49.887271 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.887274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.887325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.887433 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.888736 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.889430 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.890639 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.890866 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.890991 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.893901 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.893905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.893907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.893947 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.895335 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.895985 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.895989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.895991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.896077 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.896437 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.896441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.896442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.896643 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.896771 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.897295 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.897436 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:44:49.898518 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.885193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.3:39719)
0928 16:44:49.885743 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.885788 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.887048 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.887544 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.887551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.887997 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:49.888582 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.888598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.889580 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.889583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.890491 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.890501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.891035 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.891040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.891234 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.891944 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:49.891967 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:49.893699 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:49.895487 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.895493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.895511 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.895869 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.896176 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.896179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.896181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.896330 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.896772 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:49.896780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:49.897047 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.897239 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.897591 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":151,"threads_started":1}
I20260928 16:44:49.899742 5093 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
I20260928 16:44:49.899981 5093 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:49.900455 5093 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
I20260928 16:44:49.901484 4634 ts_manager.cc:194] Registered new tserver with Master: 474517e84526471395fbe63ddd2f46f0 (127.30.196.3:34695)
I20260928 16:44:49.902164 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:39719
I20260928 16:44:49.903797 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/test-admin.keytab.
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.921054 31504 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 16:44:49.924414 31504 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc' of type FILE
W20260928 16:44:49.924539 5100 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:49.922651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.30.196.1:40547)
0928 16:44:49.922956 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.922976 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.923003 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:44:49.923060 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.923065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.923269 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.923394 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.924277 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40142: BlockingRecv error: recv got EOF from 127.0.0.1:40142 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.925853 31504 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.927084 31504 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:49.938748 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.928229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.30.196.1:40547)
0928 16:44:49.928386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.928390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.928405 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:49.928542 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.928545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.928590 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.928666 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.929389 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.930268 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.931309 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.931870 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.931965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.934768 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.934781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.934797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.934825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.936635 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.937248 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.937254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.937255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.937350 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.937651 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.937655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.937656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.937933 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.938176 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.938338 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.938548 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 16:44:49.942919 5100 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:49.941796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.30.196.1:40547)
0928 16:44:49.941933 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.941937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.942044 (+ 107us) server_negotiation.cc:407] Connection header received
0928 16:44:49.942227 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.942235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.942276 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.942361 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.942802 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40166: BlockingRecv error: recv got EOF from 127.0.0.1:40166 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:44:49.943257 31504 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.944459 31504 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.945487 31504 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:49.954959 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.946431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.30.196.1:40547)
0928 16:44:49.946583 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.946586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.946600 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:49.946705 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.946708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.946746 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.946815 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.947309 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.947935 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.948792 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.949052 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.949138 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.951599 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.951603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.951605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.951630 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.953226 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.953735 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.953738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.953739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.953825 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.954174 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.954177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.954178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.954359 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.954464 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.954727 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.954835 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:44:49.958032 5100 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:49.956932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.30.196.1:40547)
0928 16:44:49.957069 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.957073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.957192 (+ 119us) server_negotiation.cc:407] Connection header received
0928 16:44:49.957362 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.957364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.957401 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.957482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.957921 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40184: BlockingRecv error: recv got EOF from 127.0.0.1:40184 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:44:49.958257 31504 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613859608463-31504-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.959311 31504 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.960290 31504 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:49 dist-test-slave-5rg1 krb5kdc[4589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:44:49.969111 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.961143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.30.196.1:40547)
0928 16:44:49.961290 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.961293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.961306 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:49.961411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.961413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.961450 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.961520 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.962022 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.962510 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.963271 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.963480 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.963562 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.965961 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.965964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.965966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.965990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.967283 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.967757 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.967760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.967761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.967839 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.968141 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.968143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.968145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.968308 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.968413 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.968790 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.968936 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:44:49.970424 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4696
I20260928 16:44:49.976679 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4830
I20260928 16:44:49.982456 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4964
I20260928 16:44:49.988162 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 4602
2026-09-28T16:44:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2237 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:50 dist-test-slave-5rg1 krb5kdc[5113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:50 dist-test-slave-5rg1 krb5kdc[5113](info): set up 2 sockets
Sep 28 16:44:50 dist-test-slave-5rg1 krb5kdc[5113](info): commencing operation
krb5kdc: starting...
W20260928 16:44:51.052430 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab.
Entry for principal oryx/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab.
I20260928 16:44:51.187538 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38891
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab
--principal=oryx/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:51.300598 5126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:51.300979 5126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:51.301069 5126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:51.305440 5126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:51.305558 5126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:51.305598 5126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:51.305639 5126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:51.305672 5126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:51.311244 5126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab
--principal=oryx/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:51.312511 5126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:51.313630 5126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:51.323210 5132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.323242 5134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.323210 5131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:51.323300 5126 server_base.cc:1034] running on GCE node
I20260928 16:44:51.324170 5126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:51.325343 5126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:51.326562 5126 hybrid_clock.cc:648] HybridClock initialized: now 1790613891326537 us; error 46 us; skew 500 ppm
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:51.329629 5126 init.cc:377] Logged in from keytab as oryx/127.30.196.62@KRBTEST.COM (short username oryx)
I20260928 16:44:51.330611 5126 webserver.cc:460] Webserver started at http://127.30.196.62:37653/ using document root <none> and password file <none>
I20260928 16:44:51.331197 5126 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:51.331290 5126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:51.331527 5126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:51.333776 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "023b4ece83dc4941b3bb5071f0574017"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "91e9d893e10a6f20ddfc86800fd29b08"
server_key_iv: "a888dce09e45210065b5fb34506d75d5"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.334435 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "023b4ece83dc4941b3bb5071f0574017"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "91e9d893e10a6f20ddfc86800fd29b08"
server_key_iv: "a888dce09e45210065b5fb34506d75d5"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.339282 5126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:51.341939 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.343272 5126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:51.343431 5126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "023b4ece83dc4941b3bb5071f0574017"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "91e9d893e10a6f20ddfc86800fd29b08"
server_key_iv: "a888dce09e45210065b5fb34506d75d5"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.343562 5126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:51.360934 5126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:51.364339 5126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:51.364567 5126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:51.371470 5126 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38891
I20260928 16:44:51.371546 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38891 every 8 connection(s)
I20260928 16:44:51.372488 5126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:51.374902 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5126
I20260928 16:44:51.375046 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:51.375371 31504 external_mini_cluster.cc:1428] Setting key bbc3f2b9cb20450af7d6acaa25f8b122
I20260928 16:44:51.375746 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:51.383613 5194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Bootstrap starting.
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:51.386581 5194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:51.387532 5194 log.cc:824] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:51.389683 5194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: No bootstrap required, opened a new log
I20260928 16:44:51.391392 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:51.376953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.30.196.62:38891)
0928 16:44:51.377416 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:51.377433 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:51.377474 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:44:51.378275 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:51.378297 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:51.378655 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:51.378975 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:51.380078 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.381218 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.382478 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.383097 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.383239 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:51.386427 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:51.386433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:51.386447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:51.386479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:51.388422 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.388926 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.388931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.388936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.389051 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.389348 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.389367 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.389369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.389810 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:51.390056 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:51.390354 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 16:44:51.390656 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":151,"threads_started":1}
I20260928 16:44:51.393479 5194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } }
I20260928 16:44:51.393795 5194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:51.393879 5194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023b4ece83dc4941b3bb5071f0574017, State: Initialized, Role: FOLLOWER
I20260928 16:44:51.394347 5194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } }
I20260928 16:44:51.394500 5194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:51.394572 5194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:51.394675 5194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:51.395727 5194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } }
I20260928 16:44:51.396113 5194 leader_election.cc:302] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 023b4ece83dc4941b3bb5071f0574017; no voters:
I20260928 16:44:51.396448 5194 leader_election.cc:288] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:51.396657 5199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:51.396924 5199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 LEADER]: Becoming Leader. State: Replica: 023b4ece83dc4941b3bb5071f0574017, State: Running, Role: LEADER
I20260928 16:44:51.397284 5199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } }
I20260928 16:44:51.397778 5194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:51.399701 5200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } }
I20260928 16:44:51.399971 5200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:51.399746 5201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023b4ece83dc4941b3bb5071f0574017. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } }
I20260928 16:44:51.400358 5201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:51.403529 5215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:51.403625 5215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:51.403756 5214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:51.404652 5214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:51.410653 5214 catalog_manager.cc:1347] Generated new cluster ID: 29caaffa718b4eb89e95b57edc2b8257
I20260928 16:44:51.410735 5214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:51.425065 5214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:51.426367 5214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:51.432219 5214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Generated new TSK 0
I20260928 16:44:51.432996 5214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
I20260928 16:44:51.464360 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:51.581451 5220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:51.581733 5220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:51.581777 5220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:51.585922 5220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:51.585992 5220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:51.586053 5220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:51.591475 5220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:51.592605 5220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:51.593577 5220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:51.603543 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.603498 5226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.603498 5228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:51.603767 5220 server_base.cc:1034] running on GCE node
I20260928 16:44:51.604344 5220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:51.605100 5220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:51.606335 5220 hybrid_clock.cc:648] HybridClock initialized: now 1790613891606297 us; error 50 us; skew 500 ppm
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:51.609448 5220 init.cc:377] Logged in from keytab as oryx/127.30.196.1@KRBTEST.COM (short username oryx)
I20260928 16:44:51.610476 5220 webserver.cc:460] Webserver started at http://127.30.196.1:37453/ using document root <none> and password file <none>
I20260928 16:44:51.611119 5220 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:51.611214 5220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:51.611459 5220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:51.613652 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "8f5777e0b099453aab44c047d045e60b"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "03438ee9b8530fc3403dcc38135ea17a"
server_key_iv: "e3db79b2d58322ed7dbbb7a5eb1c50ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.614236 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5777e0b099453aab44c047d045e60b"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "03438ee9b8530fc3403dcc38135ea17a"
server_key_iv: "e3db79b2d58322ed7dbbb7a5eb1c50ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.618181 5220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:44:51.620878 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.622213 5220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:51.622383 5220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "8f5777e0b099453aab44c047d045e60b"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "03438ee9b8530fc3403dcc38135ea17a"
server_key_iv: "e3db79b2d58322ed7dbbb7a5eb1c50ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.622531 5220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:51.632543 5220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:51.635769 5220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:51.635987 5220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:51.636626 5220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:51.637506 5220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:51.637579 5220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.637656 5220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:51.637701 5220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.647500 5220 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45003
I20260928 16:44:51.647521 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45003 every 8 connection(s)
I20260928 16:44:51.648573 5220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:51.651472 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5220
I20260928 16:44:51.651607 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:51.651937 31504 external_mini_cluster.cc:1428] Setting key 2969a4c3927925e96a17e61239748b50
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:51.664841 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:51.650516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:57579 (local address 127.30.196.62:38891)
0928 16:44:51.650660 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:51.650665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:51.651470 (+ 805us) server_negotiation.cc:407] Connection header received
0928 16:44:51.653144 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:51.653148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:51.653205 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:51.653290 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:51.654955 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.655854 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.657081 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.657520 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.657667 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:51.660597 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:51.660601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:51.660604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:51.660639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:51.662438 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.662970 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.662972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.662974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.663054 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.663517 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.663520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.663521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.663797 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:51.663878 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:51.664578 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 16:44:51.664710 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.2@KRBTEST.COM" created.
I20260928 16:44:51.665997 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:51.650821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:57579)
0928 16:44:51.651314 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:51.651368 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:51.652918 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:51.653519 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:51.653528 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:51.654077 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:51.654772 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:51.654794 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.656037 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.656040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:51.656932 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:51.656943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.657689 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.657692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:51.657914 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:51.658584 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:51.658601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:51.660385 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:51.662572 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:51.662578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:51.662593 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:51.662876 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:51.663170 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:51.663173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:51.663175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:51.663429 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:51.663892 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:51.663897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:51.664214 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 16:44:51.664506 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 16:44:51.664810 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260928 16:44:51.667517 5351 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:51.667819 5351 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:51.668365 5351 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:51.670223 5158 ts_manager.cc:194] Registered new tserver with Master: 8f5777e0b099453aab44c047d045e60b (127.30.196.1:45003)
I20260928 16:44:51.672053 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:57579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
I20260928 16:44:51.683377 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:51.797173 5356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:51.797461 5356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:51.797505 5356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:51.801841 5356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:51.801906 5356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:51.801970 5356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:51.807211 5356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:51.808370 5356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:51.809465 5356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:51.819600 5364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.819613 5362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:51.819613 5361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:51.819916 5356 server_base.cc:1034] running on GCE node
I20260928 16:44:51.820374 5356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:51.821067 5356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:51.822260 5356 hybrid_clock.cc:648] HybridClock initialized: now 1790613891822245 us; error 28 us; skew 500 ppm
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:51.825153 5356 init.cc:377] Logged in from keytab as oryx/127.30.196.2@KRBTEST.COM (short username oryx)
I20260928 16:44:51.826161 5356 webserver.cc:460] Webserver started at http://127.30.196.2:40237/ using document root <none> and password file <none>
I20260928 16:44:51.826749 5356 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:51.826822 5356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:51.826994 5356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:51.829140 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "964e20401b50486da73d8a54751fb06e"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "72a44f4d8a4c78fa69fae8d1b092b14a"
server_key_iv: "eee00ea596234c36070f9f78b67e0ea7"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.829761 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "964e20401b50486da73d8a54751fb06e"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "72a44f4d8a4c78fa69fae8d1b092b14a"
server_key_iv: "eee00ea596234c36070f9f78b67e0ea7"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.833812 5356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:51.836289 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.837724 5356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:51.837829 5356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "964e20401b50486da73d8a54751fb06e"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "72a44f4d8a4c78fa69fae8d1b092b14a"
server_key_iv: "eee00ea596234c36070f9f78b67e0ea7"
server_key_version: "encryptionkey@0"
I20260928 16:44:51.837929 5356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:51.850800 5356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:51.853960 5356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:51.854131 5356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:51.854684 5356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:51.855495 5356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:51.855543 5356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.855585 5356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:51.855600 5356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:51.864761 5356 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:45073
I20260928 16:44:51.864785 5485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:45073 every 8 connection(s)
I20260928 16:44:51.865686 5356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:51.870409 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5356
I20260928 16:44:51.870559 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:51.870875 31504 external_mini_cluster.cc:1428] Setting key 588e6567a06652d043d0c2fb9ab89b60
Sep 28 16:44:51 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:44:51.880302 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:51.867574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:45877 (local address 127.30.196.62:38891)
0928 16:44:51.867764 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:51.867769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:51.868616 (+ 847us) server_negotiation.cc:407] Connection header received
0928 16:44:51.870031 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:51.870034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:51.870093 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:51.870177 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:51.871608 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.872221 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.873423 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.873666 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.873788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:51.876442 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:51.876447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:51.876449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:51.876474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:51.877966 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.878550 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.878553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.878555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.878651 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:51.879021 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:51.879024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:51.879026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:51.879232 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:51.879347 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:51.879951 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 16:44:51.880092 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:44:51.881209 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:51.868001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:45877)
0928 16:44:51.868454 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:51.868498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:51.869815 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:51.870343 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:51.870350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:51.870800 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:51.871403 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:51.871424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.872359 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.872362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:51.873293 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:51.873304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:51.873834 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:51.873838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:51.874019 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:51.874669 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:51.874686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:51.876246 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:51.878125 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:51.878132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:51.878150 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:51.878436 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:51.878756 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:51.878759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:51.878761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:51.878920 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:51.879348 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:51.879358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:51.879639 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 16:44:51.879905 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 16:44:51.880301 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260928 16:44:51.882679 5486 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:51.883038 5486 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:51.883620 5486 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:51.884788 5158 ts_manager.cc:194] Registered new tserver with Master: 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:51.885377 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:45877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
I20260928 16:44:51.903245 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:52.020763 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:52.021083 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:52.021132 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:52.025444 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:52.025527 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:52.025593 5491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:52.031657 5491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:52.033183 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:52.034297 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.045351 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.045543 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.045394 5491 server_base.cc:1034] running on GCE node
W20260928 16:44:52.045753 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.046229 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.046952 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.048225 5491 hybrid_clock.cc:648] HybridClock initialized: now 1790613892048187 us; error 46 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.051306 5491 init.cc:377] Logged in from keytab as oryx/127.30.196.3@KRBTEST.COM (short username oryx)
I20260928 16:44:52.052309 5491 webserver.cc:460] Webserver started at http://127.30.196.3:45867/ using document root <none> and password file <none>
I20260928 16:44:52.052876 5491 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.052937 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.053110 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:52.055199 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "14b0f6cb730b0bcf307e2de62ece05b6"
server_key_iv: "be809740922c5c9c8b4f7dd0d06f2c69"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.055831 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "14b0f6cb730b0bcf307e2de62ece05b6"
server_key_iv: "be809740922c5c9c8b4f7dd0d06f2c69"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.059916 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:52.062587 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.064026 5491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:52.064157 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "14b0f6cb730b0bcf307e2de62ece05b6"
server_key_iv: "be809740922c5c9c8b4f7dd0d06f2c69"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.064275 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.083988 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.087425 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:52.087754 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.088438 5491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:52.089334 5491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:52.089381 5491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.089448 5491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:52.089486 5491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.099269 5491 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:41643
I20260928 16:44:52.099288 5620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:41643 every 8 connection(s)
I20260928 16:44:52.100323 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:52.100569 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5491
I20260928 16:44:52.100723 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:52.101075 31504 external_mini_cluster.cc:1428] Setting key 3e9adce1592121e51a5407cc04e42f9c
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:52.116565 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.102881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:37207 (local address 127.30.196.62:38891)
0928 16:44:52.103022 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.103027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.104151 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 16:44:52.105668 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.105672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.105736 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.105819 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.107102 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.107649 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.108891 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.109153 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.109270 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.112224 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.112228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.112230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.112260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.113662 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.114352 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.114357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.114359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.114500 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.114980 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.114984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.114985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.115264 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.115369 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.116189 (+ 820us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.116351 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:44:52.117537 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.103380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:37207)
0928 16:44:52.103998 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.104044 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.105420 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.106038 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.106046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.106432 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.106940 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.106955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.107844 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.107846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.108758 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.108769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.109320 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.109324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.109506 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.110163 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.110182 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.111953 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.113809 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.113816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.113835 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.114168 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.114633 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.114636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.114638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.114875 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.115405 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.115413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.115850 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.116117 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.116503 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":222,"threads_started":1}
I20260928 16:44:52.118698 5621 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:52.118963 5621 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:52.119549 5621 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:52.120643 5158 ts_manager.cc:194] Registered new tserver with Master: 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:52.121245 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:37207
I20260928 16:44:52.128207 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.196.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.61.keytab.
Entry for principal oryx/127.30.196.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.61.keytab.
I20260928 16:44:52.160790 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:35057
--webserver_interface=127.30.196.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.61.keytab
--principal=oryx/127.30.196.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38891,127.30.196.61:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:52.276292 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:52.276624 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:52.276667 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:52.281471 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:52.281563 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:52.281584 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:52.281601 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:52.281617 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:52.288010 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38891,127.30.196.61:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.61.keytab
--principal=oryx/127.30.196.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.196.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
W20260928 16:44:52.288276 5626 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.196.62:38891,127.30.196.61:35057'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 16:44:52.289259 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:52.290302 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.299568 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.299512 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.299651 5626 server_base.cc:1034] running on GCE node
W20260928 16:44:52.299518 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.300534 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.301764 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.302942 5626 hybrid_clock.cc:648] HybridClock initialized: now 1790613892302908 us; error 48 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.305936 5626 init.cc:377] Logged in from keytab as oryx/127.30.196.61@KRBTEST.COM (short username oryx)
I20260928 16:44:52.306912 5626 webserver.cc:460] Webserver started at http://127.30.196.61:36271/ using document root <none> and password file <none>
I20260928 16:44:52.307499 5626 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.307559 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.307827 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:52.310042 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/instance:
uuid: "5882f3a266364dfa9ace75c6dad1f993"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "4e08d0a6e61b37d30d3d2aebd6c716a0"
server_key_iv: "5f93d372fa3d7a9dd5c165d3c1a7067b"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.310940 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal/instance:
uuid: "5882f3a266364dfa9ace75c6dad1f993"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "4e08d0a6e61b37d30d3d2aebd6c716a0"
server_key_iv: "5f93d372fa3d7a9dd5c165d3c1a7067b"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.315420 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 16:44:52.318822 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.320513 5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:52.320691 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "5882f3a266364dfa9ace75c6dad1f993"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "4e08d0a6e61b37d30d3d2aebd6c716a0"
server_key_iv: "5f93d372fa3d7a9dd5c165d3c1a7067b"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.320834 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.335744 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.339571 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:52.339823 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.347857 5626 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.61:35057
I20260928 16:44:52.348171 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.61:35057 every 8 connection(s)
I20260928 16:44:52.348973 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:52.351773 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:52.355618 5695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:44:52.358438 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5626
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.61@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
I20260928 16:44:52.370517 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.357657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.30.196.62:38891)
0928 16:44:52.357799 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.357803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.358716 (+ 913us) server_negotiation.cc:407] Connection header received
0928 16:44:52.359961 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.359965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.360041 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.360135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.361062 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.361613 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.362647 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.362863 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.362981 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.366062 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.366068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.366077 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.366105 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.367929 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.368509 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.368513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.368515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.368604 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.369164 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.369167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.369169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.369483 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.369600 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.370196 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.370336 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:44:52.371711 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.357944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60634)
0928 16:44:52.358498 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.358580 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.359718 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.360189 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.360199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.360575 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.360912 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.360924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.361760 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.361763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.362514 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.362523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.363016 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.363020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.363327 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.363993 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.364010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.365762 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.368044 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.368054 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.368066 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.368393 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.368700 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.368704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.368706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.368969 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.369603 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.369610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.369943 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.370132 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.370561 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":110,"spinlock_wait_cycles":4224,"thread_start_us":167,"threads_started":1}
I20260928 16:44:52.371717 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.359370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.30.196.61:35057)
0928 16:44:52.359744 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.359754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.359792 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:44:52.359922 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.359937 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.360129 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.360321 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.361180 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.362058 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.362903 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.363302 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.363415 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.366373 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.366380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.366397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.366425 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.368384 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.368894 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.368912 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.368914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.369001 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.369395 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.369398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.369399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.369943 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.370240 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.370472 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.370889 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":5248,"thread_start_us":178,"threads_started":1}
I20260928 16:44:52.376983 5695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.61@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
I20260928 16:44:52.386596 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.377555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35628)
0928 16:44:52.377692 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.377707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.377788 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.378011 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.378013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.378214 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.378334 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.378340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.379090 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.379094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.379969 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.379979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.381059 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.381063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.381170 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.381766 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.381780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.383302 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.385114 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.385118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.385120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.385311 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.385592 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.385596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.385598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.385678 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.386060 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.386063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.386143 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.386259 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.386408 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:44:52.386596 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.377595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.30.196.61:35057)
0928 16:44:52.377725 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.377728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.377745 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:52.377874 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.377876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.377914 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.377981 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.378436 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.378962 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.380113 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.380912 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.381022 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.383475 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.383478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.383480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.383501 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.384985 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.385416 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.385420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.385421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.385501 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.385772 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.385775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.385777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.385967 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.386073 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.386301 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.386475 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":56}
I20260928 16:44:52.389837 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Bootstrap starting.
I20260928 16:44:52.392208 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:52.392915 5695 log.cc:824] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:52.394800 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: No bootstrap required, opened a new log
I20260928 16:44:52.397886 5695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } }
I20260928 16:44:52.398154 5695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:52.398216 5695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5882f3a266364dfa9ace75c6dad1f993, State: Initialized, Role: FOLLOWER
I20260928 16:44:52.398610 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } }
I20260928 16:44:52.399281 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } } }
I20260928 16:44:52.399411 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:52.400012 5695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:44:52.404947 5712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:52.405052 5712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:52.408675 5626 master_runner.cc:428] Detected that this master 5882f3a266364dfa9ace75c6dad1f993 is joining an existing cluster
I20260928 16:44:52.408749 5626 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.196.61:35057
I20260928 16:44:52.420044 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.411238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35642)
0928 16:44:52.411721 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.411738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.411837 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.412161 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.412166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.412444 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.412650 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.412659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.413361 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.413368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.414161 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.414170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.414775 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.414779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.414911 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.415533 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.415544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.416054 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.418330 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.418333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.418336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.418607 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.418904 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.418907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.418909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.418984 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.419425 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.419428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.419531 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.419647 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.419829 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":89,"thread_start_us":108,"threads_started":1}
I20260928 16:44:52.420081 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.411636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.30.196.61:35057)
0928 16:44:52.411793 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.411797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.411812 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:52.411985 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.411988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.412036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.412131 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.412793 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.413260 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.414290 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.414632 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.414739 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.416178 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.416181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.416184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.416207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.418203 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.418708 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.418712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.418714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.418804 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.419065 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.419068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.419070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.419330 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.419423 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.419734 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.419904 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":81}
I20260928 16:44:52.420749 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.411319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.30.196.62:38891)
0928 16:44:52.411461 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.411465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.411923 (+ 458us) server_negotiation.cc:407] Connection header received
0928 16:44:52.412037 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.412040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.412091 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.412191 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.412784 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.413601 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.415007 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.415325 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.415430 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.416733 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.416736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.416739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.416766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.418391 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.418858 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.418861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.418863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.418947 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.419269 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.419273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.419274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.419483 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.419559 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.420475 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.420596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:52.420771 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.411299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60648)
0928 16:44:52.411829 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.411841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.411922 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.412316 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.412318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.412537 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.412656 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.412661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.413719 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.413723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.414882 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.414893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.415563 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.415568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.415683 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.416130 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.416146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.416606 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.418498 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.418500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.418502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.418761 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.419046 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.419049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.419051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.419160 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.419647 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.419650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.419744 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.419853 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.419973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":5,"spinlock_wait_cycles":2816,"thread_start_us":75,"threads_started":1}
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.62@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
I20260928 16:44:52.435416 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.425591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35652)
0928 16:44:52.425956 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.426015 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.426153 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.426575 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.426583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.427097 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.427272 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.427284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.428421 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.428426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.429102 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.429111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.429569 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.429572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.429691 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.430258 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.430270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.431822 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.433677 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.433690 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.433698 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.433950 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.434252 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.434256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.434258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.434346 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.434748 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.434757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.434888 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.435042 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.435258 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260928 16:44:52.435407 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.425637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.30.196.61:35057)
0928 16:44:52.425791 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.425796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.426102 (+ 306us) server_negotiation.cc:407] Connection header received
0928 16:44:52.426292 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.426295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.426352 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.426429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.427422 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.428258 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.429235 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.429457 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.429565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.432001 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.432005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.432007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.432032 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.433554 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.434056 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.434061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.434063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.434156 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.434438 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.434443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.434445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.434638 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.434747 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.435095 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.435232 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for oryx/127.30.196.60@KRBTEST.COM; defaulting to no policy
I20260928 16:44:52.437352 5158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.196.61:35057: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993"
member_type: NON_VOTER
last_known_addr {
host: "127.30.196.61"
port: 35057
}
attrs {
promote: true
}
}
dest_uuid: "023b4ece83dc4941b3bb5071f0574017"
cas_config_opid_index: -1
I20260928 16:44:52.438197 5158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.60@KRBTEST.COM" created.
I20260928 16:44:52.440241 5724 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } }
I20260928 16:44:52.441634 5668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:52.443008 5724 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.196.61:35057
I20260928 16:44:52.443092 5668 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 16:44:52.443459 5626 master.cc:537] Master@127.30.196.61:35057 shutting down...
I20260928 16:44:52.443472 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } }
I20260928 16:44:52.443583 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:52.444156 5723 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:44:52.446290 5144 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 16:44:52.447203 5144 consensus_peers.cc:487] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 -> Peer 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61:35057): Couldn't send request to peer 5882f3a266364dfa9ace75c6dad1f993. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:52.448321 5626 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:52.448621 5626 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:52.448745 5626 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: stopping tablet replica
I20260928 16:44:52.463351 5626 master.cc:552] Master@127.30.196.61:35057 shutdown complete.
I20260928 16:44:52.463420 5626 master_runner.cc:305] Clearing existing system tablet
I20260928 16:44:52.463773 5626 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.463835 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.465237 5626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.60.keytab.
Entry for principal oryx/127.30.196.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.60.keytab.
I20260928 16:44:52.466167 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.466349 5626 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.466459 5626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.466491 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "5882f3a266364dfa9ace75c6dad1f993"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "4e08d0a6e61b37d30d3d2aebd6c716a0"
server_key_iv: "5f93d372fa3d7a9dd5c165d3c1a7067b"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.466920 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:38201
--webserver_interface=127.30.196.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.60.keytab
--principal=oryx/127.30.196.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:52.468783 5626 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:52.471503 5626 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:44:52.471635 5626 log.cc:1196] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 16:44:52.472045 5626 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Deleting consensus metadata
I20260928 16:44:52.472254 5626 master_runner.cc:315] Copying system tablet from 127.30.196.62:38891
I20260928 16:44:52.473704 5626 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: tablet copy: Beginning tablet copy session from remote peer at address 127.30.196.62:38891
I20260928 16:44:52.483979 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.474440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60654)
0928 16:44:52.474757 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.474772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.474891 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.475254 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.475257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.475471 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.475620 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.475627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.476891 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.476894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.477823 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.477833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.478543 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.478554 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.478657 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.479438 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.479453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.479971 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.482201 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.482205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.482207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.482482 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.482791 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.482794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.482796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.482891 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.483356 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.483359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.483467 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.483578 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.483765 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260928 16:44:52.483995 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.474574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.30.196.62:38891)
0928 16:44:52.474712 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.474716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.474867 (+ 151us) server_negotiation.cc:407] Connection header received
0928 16:44:52.475001 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.475006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.475079 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.475202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.475790 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.476738 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.477967 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.478401 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.478538 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.480140 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.480144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.480147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.480178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.482066 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.482592 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.482595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.482598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.482692 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.483003 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.483006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.483008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.483241 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.483341 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.483661 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.483822 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:44:52.485090 5178 tablet_copy_service.cc:140] P 023b4ece83dc4941b3bb5071f0574017: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5882f3a266364dfa9ace75c6dad1f993 ({username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:60654)
I20260928 16:44:52.485297 5178 tablet_copy_service.cc:161] P 023b4ece83dc4941b3bb5071f0574017: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5882f3a266364dfa9ace75c6dad1f993 at {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:60654: session id = 5882f3a266364dfa9ace75c6dad1f993-00000000000000000000000000000000
I20260928 16:44:52.487295 5178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:44:52.489427 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:44:52.492182 5626 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: tablet copy: Starting download of 0 data blocks...
I20260928 16:44:52.492390 5626 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: tablet copy: Starting download of 1 WAL segments...
I20260928 16:44:52.494375 5626 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:44:52.496402 5178 tablet_copy_service.cc:342] P 023b4ece83dc4941b3bb5071f0574017: Request end of tablet copy session 5882f3a266364dfa9ace75c6dad1f993-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:60654
I20260928 16:44:52.496548 5178 tablet_copy_service.cc:433] P 023b4ece83dc4941b3bb5071f0574017: ending tablet copy session 5882f3a266364dfa9ace75c6dad1f993-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5882f3a266364dfa9ace75c6dad1f993
I20260928 16:44:52.504720 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.506592 5737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.506610 5740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.506909 5626 server_base.cc:1034] running on GCE node
W20260928 16:44:52.511873 5738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.512226 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.512614 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.513754 5626 hybrid_clock.cc:648] HybridClock initialized: now 1790613892513731 us; error 47 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.515443 5626 init.cc:377] Logged in from keytab as oryx/127.30.196.61@KRBTEST.COM (short username oryx)
I20260928 16:44:52.516002 5626 webserver.cc:460] Webserver started at http://127.30.196.61:35323/ using document root <none> and password file <none>
I20260928 16:44:52.516207 5626 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.516245 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.517580 5626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.518570 5747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.518774 5626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.518827 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "5882f3a266364dfa9ace75c6dad1f993"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "4e08d0a6e61b37d30d3d2aebd6c716a0"
server_key_iv: "5f93d372fa3d7a9dd5c165d3c1a7067b"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.518903 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.538877 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.541118 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.546285 5626 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.61:35057
I20260928 16:44:52.546422 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.61:35057 every 8 connection(s)
I20260928 16:44:52.546916 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:52.548858 5800 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:44:52.549984 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Bootstrap starting.
I20260928 16:44:52.560737 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:44:52.561098 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Bootstrap complete.
I20260928 16:44:52.561708 5800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } }
I20260928 16:44:52.561866 5800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5882f3a266364dfa9ace75c6dad1f993, State: Initialized, Role: LEARNER
I20260928 16:44:52.562209 5800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } }
I20260928 16:44:52.562597 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } }
I20260928 16:44:52.562693 5800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:52.562789 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:52.566349 5814 catalog_manager.cc:1259] Loaded cluster ID: 29caaffa718b4eb89e95b57edc2b8257
I20260928 16:44:52.566416 5814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: loading cluster ID for follower catalog manager: success
I20260928 16:44:52.568557 5814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: acquiring CA information for follower catalog manager: success
I20260928 16:44:52.569887 5814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:44:52.609297 5730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:52.609651 5730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:52.609748 5730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:52.614091 5730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:52.614262 5730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:52.614359 5730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:52.614416 5730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:52.614454 5730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:52.620559 5730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.60.keytab
--principal=oryx/127.30.196.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:38201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.196.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:52.621759 5730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:52.622805 5730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.632282 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.632263 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.632261 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.632661 5730 server_base.cc:1034] running on GCE node
I20260928 16:44:52.633267 5730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.634644 5730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.635833 5730 hybrid_clock.cc:648] HybridClock initialized: now 1790613892635796 us; error 59 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.639041 5730 init.cc:377] Logged in from keytab as oryx/127.30.196.60@KRBTEST.COM (short username oryx)
I20260928 16:44:52.640113 5730 webserver.cc:460] Webserver started at http://127.30.196.60:33129/ using document root <none> and password file <none>
I20260928 16:44:52.640748 5730 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.640844 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.641088 5730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:52.643491 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/instance:
uuid: "763a5cae2d414b0dbf4f583edd06c602"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "f9c82d7b1c7e7e84daf79d637602ef42"
server_key_iv: "4fd92b61e47e3a5c247d5c95f0847b12"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.644266 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal/instance:
uuid: "763a5cae2d414b0dbf4f583edd06c602"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "f9c82d7b1c7e7e84daf79d637602ef42"
server_key_iv: "4fd92b61e47e3a5c247d5c95f0847b12"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.648554 5730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:52.651162 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.652602 5730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:44:52.652755 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "763a5cae2d414b0dbf4f583edd06c602"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "f9c82d7b1c7e7e84daf79d637602ef42"
server_key_iv: "4fd92b61e47e3a5c247d5c95f0847b12"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.652876 5730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.672572 5730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.674661 5351 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
I20260928 16:44:52.676070 5730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:52.676285 5730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.684006 5730 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.60:38201
I20260928 16:44:52.684376 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.60:38201 every 8 connection(s)
I20260928 16:44:52.685055 5730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:52.687772 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5730
I20260928 16:44:52.688048 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:52.692600 5883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.60@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:52.704130 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.689632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.30.196.60:38201)
0928 16:44:52.690059 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.690074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.690106 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:44:52.690709 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.690725 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.691045 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.691371 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.692221 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.693286 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.694352 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.695040 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.695201 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.698309 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.698317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.698371 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.698407 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.700659 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.701146 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.701152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.701158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.701289 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.701662 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.701665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.701667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.702274 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.702503 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.702751 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.703051 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":153,"threads_started":1}
I20260928 16:44:52.704653 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.694435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.30.196.62:38891)
0928 16:44:52.694566 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.694570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.694793 (+ 223us) server_negotiation.cc:407] Connection header received
0928 16:44:52.694920 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.694923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.694984 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.695094 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.695998 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.696494 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.697577 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.697968 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.698073 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.701037 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.701040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.701042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.701066 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.702804 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.703356 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.703359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.703360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.703442 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.703754 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.703756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.703757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.703929 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.704007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.704412 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.704535 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:52.705112 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.694277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60656)
0928 16:44:52.694665 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.694711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.694827 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.695169 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.695174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.695626 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.695871 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.695881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.696639 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.696644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.697458 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.697468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.698136 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.698140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.698277 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.698990 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.699046 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.700832 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.702924 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.702930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.702935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.703251 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.703537 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.703540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.703542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.703630 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.704038 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.704043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.704163 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.704812 (+ 649us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.704977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":151,"threads_started":1}
I20260928 16:44:52.706336 5883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.60@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
I20260928 16:44:52.714800 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.706853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35656)
0928 16:44:52.707034 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.707046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.707109 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.707537 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.707540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.707747 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.707839 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.707845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.708757 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.708760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.709323 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.709331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.709806 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.709809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.709904 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.710369 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.710380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.711782 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.713417 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.713420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.713422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.713634 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.713933 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.713935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.713936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.714008 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.714379 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.714382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.714466 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.714557 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.714673 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:44:52.715135 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.707062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.30.196.61:35057)
0928 16:44:52.707329 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.707332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.707343 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:44:52.707380 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.707382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.707419 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.707525 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.707954 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.708638 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.709442 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.709696 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.709807 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.711914 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.711916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.711919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.711944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.713308 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.713740 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.713743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.713744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.713839 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.714086 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.714089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.714090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.714271 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.714349 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.714697 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.714806 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260928 16:44:52.715934 5883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.60@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
I20260928 16:44:52.724280 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.716407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60324)
0928 16:44:52.716555 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.716571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.716644 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.716825 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.716827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.716959 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.717030 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.717034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.717730 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.717732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.718245 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.718253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.719402 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.719405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.719469 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.719996 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.720011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.721258 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.722854 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.722857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.722859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.723048 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.723318 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.723320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.723321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.723395 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.723830 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.723832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.723912 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.723998 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.724131 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:44:52.724318 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.716476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.30.196.60:38201)
0928 16:44:52.716610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.716613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.716625 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:44:52.716702 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.716704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.716737 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.716798 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.717140 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.717637 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.718373 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.719280 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.719378 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.721382 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.721385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.721387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.721409 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.722740 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.723154 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.723157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.723158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.723231 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.723483 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.723485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.723487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.723729 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.723822 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.724067 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.724193 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":66}
I20260928 16:44:52.727360 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Bootstrap starting.
I20260928 16:44:52.729804 5883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:52.730584 5883 log.cc:824] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:52.732774 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: No bootstrap required, opened a new log
I20260928 16:44:52.735946 5883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } }
I20260928 16:44:52.736205 5883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:52.736305 5883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763a5cae2d414b0dbf4f583edd06c602, State: Initialized, Role: FOLLOWER
I20260928 16:44:52.736761 5883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } }
I20260928 16:44:52.737262 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } } }
I20260928 16:44:52.737394 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:52.738011 5883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:44:52.742825 5900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:52.742926 5900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:52.746696 5730 master_runner.cc:428] Detected that this master 763a5cae2d414b0dbf4f583edd06c602 is joining an existing cluster
I20260928 16:44:52.746789 5730 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.196.60:38201
I20260928 16:44:52.749431 5889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:52.749595 5889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } }
I20260928 16:44:52.750994 5889 leader_election.cc:288] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61:35057)
I20260928 16:44:52.750897 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "763a5cae2d414b0dbf4f583edd06c602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023b4ece83dc4941b3bb5071f0574017" is_pre_election: true
I20260928 16:44:52.751866 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "763a5cae2d414b0dbf4f583edd06c602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5882f3a266364dfa9ace75c6dad1f993" is_pre_election: true
I20260928 16:44:52.752049 5774 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Handling vote request from an unknown peer 763a5cae2d414b0dbf4f583edd06c602
I20260928 16:44:52.752188 5774 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 763a5cae2d414b0dbf4f583edd06c602 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 16:44:52.753402 5834 leader_election.cc:302] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 763a5cae2d414b0dbf4f583edd06c602; no voters: 023b4ece83dc4941b3bb5071f0574017, 5882f3a266364dfa9ace75c6dad1f993
I20260928 16:44:52.753993 5889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:52.755091 5889 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:44:52.757684 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5220
I20260928 16:44:52.764274 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.749219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.30.196.61:35057)
0928 16:44:52.749344 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.749348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.755461 (+ 6113us) server_negotiation.cc:407] Connection header received
0928 16:44:52.755520 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.755523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.755578 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.755692 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.756270 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.757071 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.758745 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.759102 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.759173 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.760700 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.760704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.760707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.760731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.762394 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.762901 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.762905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.762907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.762989 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.763282 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.763285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.763287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.763507 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.763587 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.763960 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.764103 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:44:52.764817 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.750702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35660)
0928 16:44:52.755367 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.755379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.755461 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.755736 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.755739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.756058 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.756159 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.756165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.757228 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.757232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.758609 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.758620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.759335 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.759339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.759441 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.760049 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.760064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.760545 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.762528 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.762533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.762537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.762794 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.763096 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.763098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.763100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.763186 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.763644 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.763647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.763777 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.764483 (+ 706us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.764633 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4608,"thread_start_us":92,"threads_started":1}
I20260928 16:44:52.767215 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:45003
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:38891
--webserver_port=37453
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:52.776716 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.749520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.30.196.60:38201)
0928 16:44:52.749623 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.749627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.751320 (+ 1693us) server_negotiation.cc:407] Connection header received
0928 16:44:52.751474 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.751477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.751513 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.751557 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.752209 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.752957 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.754070 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.754366 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.754441 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.773195 (+ 18754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.773200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.773202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.773224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.774941 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.775412 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.775424 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.775425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.775494 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.775800 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.775803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.775805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.776028 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.776096 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.776371 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.776530 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:44:52.777514 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.749495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60328)
0928 16:44:52.751241 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.751255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.751372 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.751716 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.751719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.751954 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.752080 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.752087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.753081 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.753084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.753950 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.753961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.754570 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.754573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.754657 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.772409 (+ 17752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.772437 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.773012 (+ 575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.775067 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.775070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.775072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.775313 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.775587 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.775589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.775591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.775709 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.776153 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.776155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.776229 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.777186 (+ 957us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.777335 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"thread_start_us":113,"threads_started":1}
I20260928 16:44:52.778362 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.749016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60662)
0928 16:44:52.750301 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.750313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.750453 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.750839 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.750841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.751104 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.751249 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.751257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.752350 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.752353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.753373 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.753387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.754097 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.754100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.754204 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.772539 (+ 18335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.772562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.773154 (+ 592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.776250 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.776257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.776261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.776589 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.776998 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.777003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.777007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.777108 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.777662 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.777667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.777797 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.777931 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.778104 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260928 16:44:52.779114 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.749002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.30.196.62:38891)
0928 16:44:52.749161 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.749165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.750395 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 16:44:52.750564 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.750567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.750622 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.750704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.751389 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.752193 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.753527 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.753830 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.753913 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.773409 (+ 19496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.773414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.773419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.773759 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.776093 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.776752 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.776758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.776761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.776860 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.777229 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.777235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.777238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.777499 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.777608 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.778711 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.778857 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.62@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
I20260928 16:44:52.792831 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.783071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60344)
0928 16:44:52.783272 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.783284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.783357 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.783757 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.783760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.783969 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.784085 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.784093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.785201 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.785204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.786064 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.786075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.786791 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.786795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.786894 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.787406 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.787433 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.789080 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.791114 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.791117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.791120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.791370 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.791716 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.791719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.791721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.791810 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.792285 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.792288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.792385 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.792483 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.792640 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 16:44:52.793488 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.783143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.30.196.60:38201)
0928 16:44:52.783273 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.783278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.783323 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:44:52.783458 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.783461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.783513 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.783589 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.784227 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.785070 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.786224 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.786577 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.786669 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.789236 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.789240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.789243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.789268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.790985 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.791499 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.791502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.791504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.791596 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.791926 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.791930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.791931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.792145 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.792231 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.793205 (+ 974us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.793327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:44:52.794559 5158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.196.60:38201: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602"
member_type: NON_VOTER
last_known_addr {
host: "127.30.196.60"
port: 38201
}
attrs {
promote: true
}
}
dest_uuid: "023b4ece83dc4941b3bb5071f0574017"
cas_config_opid_index: 5
I20260928 16:44:52.795166 5158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } }
I20260928 16:44:52.796240 5723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.796929 5857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:44:52.797469 5724 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:52.797935 5723 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.196.60:38201
I20260928 16:44:52.798141 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.798247 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:52.798795 5730 master.cc:537] Master@127.30.196.60:38201 shutting down...
W20260928 16:44:52.800915 5145 consensus_peers.cc:487] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 -> Peer 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201): Couldn't send request to peer 763a5cae2d414b0dbf4f583edd06c602. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:52.803109 5730 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:52.803385 5730 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:52.803505 5730 tablet_replica.cc:330] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: stopping tablet replica
I20260928 16:44:52.805246 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.796325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35662)
0928 16:44:52.796562 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.796576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.796647 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.801035 (+ 4388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.801039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.801065 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:52.801170 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.801176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.802269 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.802273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.803520 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.803532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.804828 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.804831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.804923 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.804969 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.805041 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.805118 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 16:44:52.805680 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.797422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.30.196.61:35057)
0928 16:44:52.798225 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.798230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.798245 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:52.798297 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.798300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.798360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.798465 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:52.801318 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.802118 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.803696 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.804493 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.804574 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.804728 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.805484 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.805570 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711}
I20260928 16:44:52.806133 5774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:44:52.806738 5910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 16:44:52.808729 5774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.811180 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023b4ece83dc4941b3bb5071f0574017. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.811344 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:52.811462 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: NON_VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: true } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.811564 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:52.812438 5910 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: attempting to promote NON_VOTER 5882f3a266364dfa9ace75c6dad1f993 to VOTER
I20260928 16:44:52.812963 5910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } }
I20260928 16:44:52.814186 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.813879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60356)
0928 16:44:52.814014 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.814099 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:44:52.814632 5774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 LEARNER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 16:44:52.815274 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.814875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60360)
0928 16:44:52.815150 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.815204 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 16:44:52.815237 5723 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 16:44:52.815400 5145 consensus_peers.cc:487] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 -> Peer 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201): Couldn't send request to peer 763a5cae2d414b0dbf4f583edd06c602. Status: Network error: Client connection negotiation failed: client connection to 127.30.196.60:38201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:52.817085 5910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.817628 5774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.818643 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.818745 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:52.818871 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.818936 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:52.819070 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.819171 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:52.820497 5730 master.cc:552] Master@127.30.196.60:38201 shutdown complete.
I20260928 16:44:52.820555 5730 master_runner.cc:305] Clearing existing system tablet
I20260928 16:44:52.820806 5730 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.820869 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.822242 5730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 16:44:52.823187 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.823411 5730 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.823544 5730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.823576 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "763a5cae2d414b0dbf4f583edd06c602"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "f9c82d7b1c7e7e84daf79d637602ef42"
server_key_iv: "4fd92b61e47e3a5c247d5c95f0847b12"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.825774 5730 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:52.828718 5730 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:44:52.828819 5730 log.cc:1196] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 16:44:52.829092 5730 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Deleting consensus metadata
I20260928 16:44:52.829270 5730 master_runner.cc:315] Copying system tablet from 127.30.196.62:38891
I20260928 16:44:52.830461 5730 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: tablet copy: Beginning tablet copy session from remote peer at address 127.30.196.62:38891
I20260928 16:44:52.840157 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.831143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.30.196.62:38891)
0928 16:44:52.831277 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.831282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.831438 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:44:52.831562 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:52.831564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:52.831617 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:52.831750 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:52.832273 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.833199 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.834423 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.834750 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.834860 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.836241 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:52.836245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:52.836248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:52.836277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:52.838119 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.838728 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.838732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.838734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.838835 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:52.839142 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:52.839145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:52.839147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:52.839378 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:52.839490 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:52.839848 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 16:44:52.839982 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:52.840764 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.831024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:60664)
0928 16:44:52.831332 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.831346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:52.831463 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:52.831765 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:52.831768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:52.832004 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:52.832142 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.832149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.833330 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.833333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.834277 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:52.834288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:52.834883 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:52.834887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:52.834988 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:52.835606 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:52.835621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:52.836087 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:52.838260 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.838263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.838265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.838555 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.838947 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:52.838951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:52.838953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:52.839053 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:52.839503 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:52.839506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:52.839606 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:44:52.840404 (+ 798us) client_negotiation.cc:240] Negotiation successful
0928 16:44:52.840573 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260928 16:44:52.841264 5178 tablet_copy_service.cc:140] P 023b4ece83dc4941b3bb5071f0574017: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 763a5cae2d414b0dbf4f583edd06c602 ({username='oryx', principal='oryx/127.30.196.60@KRBTEST.COM'} at 127.0.0.1:60664)
I20260928 16:44:52.841369 5178 tablet_copy_service.cc:161] P 023b4ece83dc4941b3bb5071f0574017: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 763a5cae2d414b0dbf4f583edd06c602 at {username='oryx', principal='oryx/127.30.196.60@KRBTEST.COM'} at 127.0.0.1:60664: session id = 763a5cae2d414b0dbf4f583edd06c602-00000000000000000000000000000000
I20260928 16:44:52.842509 5178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:44:52.843943 5730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:44:52.846388 5730 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: tablet copy: Starting download of 0 data blocks...
I20260928 16:44:52.846675 5730 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: tablet copy: Starting download of 1 WAL segments...
I20260928 16:44:52.848455 5730 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:44:52.850401 5178 tablet_copy_service.cc:342] P 023b4ece83dc4941b3bb5071f0574017: Request end of tablet copy session 763a5cae2d414b0dbf4f583edd06c602-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.196.60@KRBTEST.COM'} at 127.0.0.1:60664
I20260928 16:44:52.850502 5178 tablet_copy_service.cc:433] P 023b4ece83dc4941b3bb5071f0574017: ending tablet copy session 763a5cae2d414b0dbf4f583edd06c602-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 763a5cae2d414b0dbf4f583edd06c602
I20260928 16:44:52.858685 5730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.860769 5924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.860792 5921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.860994 5922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.863914 5730 server_base.cc:1034] running on GCE node
I20260928 16:44:52.864117 5730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.864476 5730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.865626 5730 hybrid_clock.cc:648] HybridClock initialized: now 1790613892865611 us; error 33 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.867355 5730 init.cc:377] Logged in from keytab as oryx/127.30.196.60@KRBTEST.COM (short username oryx)
I20260928 16:44:52.867929 5730 webserver.cc:460] Webserver started at http://127.30.196.60:45255/ using document root <none> and password file <none>
I20260928 16:44:52.868131 5730 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.868180 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.869665 5730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.870796 5931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.871125 5730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:52.871213 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "763a5cae2d414b0dbf4f583edd06c602"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "f9c82d7b1c7e7e84daf79d637602ef42"
server_key_iv: "4fd92b61e47e3a5c247d5c95f0847b12"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.871305 5730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.879631 5730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.882130 5730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.887254 5730 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.60:38201
I20260928 16:44:52.887290 5984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.60:38201 every 8 connection(s)
I20260928 16:44:52.887535 5486 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
I20260928 16:44:52.887889 5730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:52.889835 5985 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:44:52.890908 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Bootstrap starting.
I20260928 16:44:52.903010 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:44:52.903373 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: Bootstrap complete.
I20260928 16:44:52.904047 5985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } }
I20260928 16:44:52.904212 5985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 763a5cae2d414b0dbf4f583edd06c602, State: Initialized, Role: LEARNER
I20260928 16:44:52.904341 5985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } }
I20260928 16:44:52.904759 5988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:52.904888 5985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:52.904989 5988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:52.908536 5999 catalog_manager.cc:1259] Loaded cluster ID: 29caaffa718b4eb89e95b57edc2b8257
I20260928 16:44:52.908681 5999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: loading cluster ID for follower catalog manager: success
I20260928 16:44:52.910928 5999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: acquiring CA information for follower catalog manager: success
I20260928 16:44:52.912431 5999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:44:52.922163 5909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:52.922446 5909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:52.922501 5909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:52.926802 5909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:52.926875 5909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:52.926982 5909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:52.932657 5909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:45003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=37453
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:52.933956 5909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:52.935006 5909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:52.945299 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:52.945174 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.945317 5909 server_base.cc:1034] running on GCE node
W20260928 16:44:52.945174 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:52.945899 5909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:52.946559 5909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:52.947741 5909 hybrid_clock.cc:648] HybridClock initialized: now 1790613892947704 us; error 48 us; skew 500 ppm
Sep 28 16:44:52 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:52.950608 5909 init.cc:377] Logged in from keytab as oryx/127.30.196.1@KRBTEST.COM (short username oryx)
I20260928 16:44:52.951526 5909 webserver.cc:460] Webserver started at http://127.30.196.1:37453/ using document root <none> and password file <none>
I20260928 16:44:52.952190 5909 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:52.952317 5909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:52.956298 5909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:52.958390 6014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.959645 5909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:52.959864 5909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "8f5777e0b099453aab44c047d045e60b"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "03438ee9b8530fc3403dcc38135ea17a"
server_key_iv: "e3db79b2d58322ed7dbbb7a5eb1c50ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:52.960436 5909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:52.978899 5909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:52.982411 5909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:52.982662 5909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:52.983312 5909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:52.984292 5909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:52.984370 5909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.984443 5909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:52.984503 5909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:52.994469 5909 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45003
I20260928 16:44:52.994505 6127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45003 every 8 connection(s)
I20260928 16:44:52.995626 5909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:53.005177 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 5909
I20260928 16:44:53.005383 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5356
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
I20260928 16:44:53.014237 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:45073
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:38891
--webserver_port=40237
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613892, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:53.015529 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.997645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:59675 (local address 127.30.196.61:35057)
0928 16:44:52.997770 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.997774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.999232 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 16:44:53.000455 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.000459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.000529 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.000616 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.003080 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.003884 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.005126 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.005480 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.005598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.010264 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.010271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.010278 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.010315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.012262 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013162 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.013166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.013171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.013278 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013742 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.013745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.013747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.013986 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.014102 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.015226 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.015368 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:53.015993 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.999654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:46925 (local address 127.30.196.60:38201)
0928 16:44:52.999924 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.999928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.999940 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:44:53.000563 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.000569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.000625 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.000751 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.003799 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.004993 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.006205 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.006523 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.006641 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.010720 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.010726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.010733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.010770 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.012836 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013423 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.013426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.013428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.013524 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013875 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.013878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.013880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.014125 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.014228 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.015235 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.015407 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260928 16:44:53.016731 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.999286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.30.196.1:46925)
0928 16:44:52.999509 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.999519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.000228 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.000833 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.000842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.001458 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.002169 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.002179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.005218 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.005222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.006093 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.006106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.006654 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.006660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.006807 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.007390 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.007411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.010040 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.012995 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.013006 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.013296 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.013627 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.013638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.013746 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.014276 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.014279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.014694 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.014809 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.015393 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":180,"spinlock_wait_cycles":26496,"thread_start_us":92,"threads_started":1}
I20260928 16:44:53.017343 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.998063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.30.196.1:59675)
0928 16:44:52.999106 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.999121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.000235 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.000833 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.000842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.001456 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.002171 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.002179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.004006 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.004010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.004995 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.005004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.005789 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.005796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.006060 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.007122 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.007151 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.010040 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.012517 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.012523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.012541 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.013008 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.013397 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.013400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.013402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.013622 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.014169 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.014179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.014563 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.014796 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.017138 (+ 2342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":2050,"spinlock_wait_cycles":77696,"thread_start_us":171,"threads_started":1}
I20260928 16:44:53.018615 6130 heartbeater.cc:347] Connected to a master server at 127.30.196.61:35057
I20260928 16:44:53.018878 6128 heartbeater.cc:347] Connected to a master server at 127.30.196.60:38201
I20260928 16:44:53.018996 6128 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.019730 6128 heartbeater.cc:511] Master 127.30.196.60:38201 requested a full tablet report, sending...
I20260928 16:44:53.018900 6130 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.020016 6130 heartbeater.cc:511] Master 127.30.196.61:35057 requested a full tablet report, sending...
I20260928 16:44:53.021559 5948 ts_manager.cc:194] Registered new tserver with Master: 8f5777e0b099453aab44c047d045e60b (127.30.196.1:45003)
I20260928 16:44:53.021987 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.998063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:50161)
0928 16:44:52.998692 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:52.998762 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.000228 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.000833 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.000843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.001451 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.002158 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.002175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.003053 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.003057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.004364 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.004374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.012721 (+ 8347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.012725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.012856 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.013393 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.013407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.015214 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.020352 (+ 5138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.020355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.020358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.020616 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.020944 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.020946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.020947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.021015 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.021421 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.021424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.021527 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.021634 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.021801 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":103,"spinlock_wait_cycles":17024,"thread_start_us":739,"threads_started":1}
I20260928 16:44:53.022006 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:52.997623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:50161 (local address 127.30.196.62:38891)
0928 16:44:52.997770 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:52.997775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:52.998922 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 16:44:53.000468 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.000471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.000537 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.000592 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.002335 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.002893 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.004495 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.004727 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.004848 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.017595 (+ 12747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.017599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.017602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.017638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.019626 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.020748 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.020751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.020753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.020850 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.021124 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.021127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.021129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.021317 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.021432 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.021741 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.021867 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:44:53.022720 5764 ts_manager.cc:194] Registered new tserver with Master: 8f5777e0b099453aab44c047d045e60b (127.30.196.1:45003)
I20260928 16:44:53.022779 6129 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:53.022864 6129 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.023032 6129 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:53.023913 5158 ts_manager.cc:194] Re-registered known tserver with Master: 8f5777e0b099453aab44c047d045e60b (127.30.196.1:45003)
I20260928 16:44:53.024497 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:50161
I20260928 16:44:53.123301 5621 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
W20260928 16:44:53.133774 6136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:53.134065 6136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:53.134151 6136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:53.138427 6136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:53.138506 6136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:53.138603 6136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:53.144291 6136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:45073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=40237
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:53.145510 6136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:53.146696 6136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:53.157361 6141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:53.157429 6136 server_base.cc:1034] running on GCE node
W20260928 16:44:53.157368 6144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:53.157357 6142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:53.158095 6136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:53.158728 6136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:53.159978 6136 hybrid_clock.cc:648] HybridClock initialized: now 1790613893159950 us; error 41 us; skew 500 ppm
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:53.163189 6136 init.cc:377] Logged in from keytab as oryx/127.30.196.2@KRBTEST.COM (short username oryx)
I20260928 16:44:53.164314 6136 webserver.cc:460] Webserver started at http://127.30.196.2:40237/ using document root <none> and password file <none>
I20260928 16:44:53.165098 6136 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:53.165210 6136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:53.169353 6136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:53.171832 6151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.173318 6136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:53.173475 6136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "964e20401b50486da73d8a54751fb06e"
format_stamp: "Formatted at 2026-09-28 16:44:51 on dist-test-slave-5rg1"
server_key: "72a44f4d8a4c78fa69fae8d1b092b14a"
server_key_iv: "eee00ea596234c36070f9f78b67e0ea7"
server_key_version: "encryptionkey@0"
I20260928 16:44:53.174118 6136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:53.193454 6136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:53.196854 6136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:53.197093 6136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:53.197783 6136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:53.198730 6136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:53.198804 6136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.198876 6136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:53.198936 6136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.208648 6136 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:45073
I20260928 16:44:53.208686 6264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:45073 every 8 connection(s)
I20260928 16:44:53.209697 6136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:53.212419 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6136
I20260928 16:44:53.212617 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5491
I20260928 16:44:53.221701 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:41643
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:38891
--webserver_port=45867
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
I20260928 16:44:53.229396 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.211593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:52721 (local address 127.30.196.62:38891)
0928 16:44:53.211907 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.211911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.214354 (+ 2443us) server_negotiation.cc:407] Connection header received
0928 16:44:53.214456 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.214459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.214515 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.214581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.216563 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.217380 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.219413 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.219782 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.219860 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.224356 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.224360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.224363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.224393 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.226558 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227330 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.227334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.227336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.227430 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227914 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.227917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.227919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.228146 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.228233 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.229091 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.229223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260928 16:44:53.229396 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.211615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:53289 (local address 127.30.196.61:35057)
0928 16:44:53.211808 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.211813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.213257 (+ 1444us) server_negotiation.cc:407] Connection header received
0928 16:44:53.214463 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.214466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.214522 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.214576 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.216568 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.217380 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.219412 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.219782 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.219860 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.224356 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.224360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.224363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.224393 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.226506 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227330 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.227334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.227336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.227426 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227914 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.227918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.227919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.228156 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.228233 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.229091 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.229223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:44:53.231144 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.212053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:52721)
0928 16:44:53.214226 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.214244 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.214361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.214773 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.214782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.215479 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.216390 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.216410 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.217549 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.217552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.219270 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.219281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.220065 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.220069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.220375 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.221232 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.221261 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.224146 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.226678 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.226685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.226705 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.227206 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.227546 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.227551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.227799 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.228305 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.228312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.228770 (+ 458us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.229579 (+ 809us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.230078 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2064,"mutex_wait_us":560,"spinlock_wait_cycles":81536,"thread_start_us":180,"threads_started":1}
I20260928 16:44:53.231144 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.212053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.30.196.2:53289)
0928 16:44:53.213103 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.213158 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.214220 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.214771 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.214778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.215479 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.216390 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.216410 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.217552 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.217555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.219272 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.219281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.220065 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.220069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.220372 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.221232 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.221261 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.224146 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.226678 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.226685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.226705 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.227206 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.227546 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.227549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.227551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.227799 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.228305 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.228312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.228770 (+ 458us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.229670 (+ 900us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.230027 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":430,"spinlock_wait_cycles":46080,"thread_start_us":137,"threads_started":1}
I20260928 16:44:53.232065 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.213487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33833 (local address 127.30.196.60:38201)
0928 16:44:53.213631 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.213636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.219120 (+ 5484us) server_negotiation.cc:407] Connection header received
0928 16:44:53.219305 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.219311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.219394 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.219453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.220388 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.221239 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.222487 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.222887 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.223076 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.226570 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.226576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.226582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.226618 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.228621 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.229895 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.229901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.229904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.230008 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.230395 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.230401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.230405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.230667 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.230768 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.231195 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.231350 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:44:53.232551 6266 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:53.232746 6267 heartbeater.cc:347] Connected to a master server at 127.30.196.61:35057
I20260928 16:44:53.232821 6267 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.233536 6267 heartbeater.cc:511] Master 127.30.196.61:35057 requested a full tablet report, sending...
I20260928 16:44:53.232821 6266 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.234364 6266 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:53.232812 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.213144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.30.196.2:33833)
0928 16:44:53.219022 (+ 5878us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.219040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.219195 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.219684 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.219690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.220065 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.220230 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.220241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.221396 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.221398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.222311 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.222332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.223272 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.223278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.223392 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.223998 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.224019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.226380 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.229023 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.229029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.229032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.229337 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.230162 (+ 825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.230168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.230172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.230274 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.230837 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.230843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.230959 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.232297 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.232482 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5758,"thread_start_us":92,"threads_started":1}
I20260928 16:44:53.234856 5764 ts_manager.cc:194] Registered new tserver with Master: 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:53.235106 5158 ts_manager.cc:194] Re-registered known tserver with Master: 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:53.235612 6265 heartbeater.cc:347] Connected to a master server at 127.30.196.60:38201
I20260928 16:44:53.235744 6265 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.235919 6265 heartbeater.cc:511] Master 127.30.196.60:38201 requested a full tablet report, sending...
I20260928 16:44:53.235942 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:52721
I20260928 16:44:53.236627 5948 ts_manager.cc:194] Registered new tserver with Master: 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:53.309880 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.305550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60362)
0928 16:44:53.305688 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.305705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.305849 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.306227 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.306230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.306248 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.306340 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.306345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.307549 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.307554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.308407 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.308428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.309526 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.309529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.309603 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.309626 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.309683 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.309754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 16:44:53.309929 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.305709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.30.196.60:38201)
0928 16:44:53.305899 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.305905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.305922 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:53.305979 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.305982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.306049 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.306181 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.306473 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.307345 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.308565 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.309375 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.309499 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.309679 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.309721 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.309797 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 16:44:53.310716 5958 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 16:44:53.311151 5988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023b4ece83dc4941b3bb5071f0574017. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:53.311337 5988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: LEARNER
W20260928 16:44:53.345839 6272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:53.346128 6272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:53.346174 6272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:53.350666 6272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:53.350742 6272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:53.350805 6272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:53.356601 6272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:41643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=45867
--tserver_master_addrs=127.30.196.62:38891,127.30.196.61:35057,127.30.196.60:38201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:53.357722 6272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:53.358717 6272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:53.369649 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:53.369647 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:53.369661 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:53.369927 6272 server_base.cc:1034] running on GCE node
I20260928 16:44:53.370527 6272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:53.371223 6272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:53.372437 6272 hybrid_clock.cc:648] HybridClock initialized: now 1790613893372420 us; error 39 us; skew 500 ppm
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:53.375888 6272 init.cc:377] Logged in from keytab as oryx/127.30.196.3@KRBTEST.COM (short username oryx)
I20260928 16:44:53.377027 6272 webserver.cc:460] Webserver started at http://127.30.196.3:45867/ using document root <none> and password file <none>
I20260928 16:44:53.377738 6272 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:53.377836 6272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:53.382956 6272 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:44:53.385461 6288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.386866 6272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:53.386977 6272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
format_stamp: "Formatted at 2026-09-28 16:44:52 on dist-test-slave-5rg1"
server_key: "14b0f6cb730b0bcf307e2de62ece05b6"
server_key_iv: "be809740922c5c9c8b4f7dd0d06f2c69"
server_key_version: "encryptionkey@0"
I20260928 16:44:53.387620 6272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:53.404665 6272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:53.408018 6272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:53.408231 6272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:53.408875 6272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:53.409927 6272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:53.409974 6272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.410015 6272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:53.410068 6272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:53.419653 6272 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:41643
I20260928 16:44:53.419744 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:41643 every 8 connection(s)
I20260928 16:44:53.420769 6272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:53.430054 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6272
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.60@KRBTEST.COM
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.61@KRBTEST.COM
Sep 28 16:44:53 dist-test-slave-5rg1 krb5kdc[5113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613893, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:53.437608 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.424201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:40015 (local address 127.30.196.60:38201)
0928 16:44:53.424329 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.424333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.424421 (+ 88us) server_negotiation.cc:407] Connection header received
0928 16:44:53.425334 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.425337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.425394 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.425482 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.426731 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.427532 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.428872 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429232 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429331 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.433864 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.433869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.433874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.433905 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.435689 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.436176 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.436179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.436181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.436258 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.436570 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.436573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.436574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.436772 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.436864 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.437312 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.437451 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:53.438498 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.424066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.30.196.3:40015)
0928 16:44:53.424313 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.424326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.424819 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.425511 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.425514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.425909 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.426571 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.426580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.427655 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.427658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.428727 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.428737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429373 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.429558 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.430339 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.430350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.433600 (+ 3250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.435803 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.435806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.435808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.436068 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.436361 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.436364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.436366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.436468 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.436906 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.436910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.436981 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.437094 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.437907 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":740,"spinlock_wait_cycles":23680,"thread_start_us":87,"threads_started":1}
I20260928 16:44:53.438715 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.422572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55205 (local address 127.30.196.62:38891)
0928 16:44:53.422729 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.422734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.423931 (+ 1197us) server_negotiation.cc:407] Connection header received
0928 16:44:53.425072 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.425076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.425131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.425214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.426731 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.427532 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429106 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429417 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429490 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.434705 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.434711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.434717 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.434749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.436533 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.437010 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.437013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.437015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.437097 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.437383 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.437389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.437394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.437615 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.437694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.438137 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.438262 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:53.438782 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.423010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.30.196.3:46419)
0928 16:44:53.423774 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.423828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.424818 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.425373 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.425381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.425923 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.426562 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.426577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.427413 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.427419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.428757 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.428767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429481 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.429586 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.430262 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.430283 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.432794 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.435061 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.435070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.435092 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.435439 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.435803 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.435806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.435808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.436032 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.436542 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.436552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.436878 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.437094 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.437450 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":122,"spinlock_wait_cycles":3712,"thread_start_us":122,"threads_started":1}
I20260928 16:44:53.439266 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.423010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:55205)
0928 16:44:53.423774 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.423828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.424819 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.425371 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.425378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.425909 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:53.426571 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.426580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.428164 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.428167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.428986 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.428994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429641 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.429751 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.432973 (+ 3222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:53.432995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:53.434534 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:53.436663 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.436665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.436667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.436898 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.437195 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:53.437199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:53.437201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:53.437279 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:53.437829 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:53.437832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:53.437931 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.438906 (+ 975us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.439055 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":140,"spinlock_wait_cycles":19200,"thread_start_us":150,"threads_started":1}
I20260928 16:44:53.439968 6403 heartbeater.cc:347] Connected to a master server at 127.30.196.60:38201
I20260928 16:44:53.440140 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.422533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:46419 (local address 127.30.196.61:35057)
0928 16:44:53.422682 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.422687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.423931 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 16:44:53.425073 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.425076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.425131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.425191 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.426719 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.427256 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.428911 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.429231 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.429302 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.432997 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.433000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.433003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.433032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.434861 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.435572 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.435575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.435578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.435698 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.436153 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.436159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.436161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.436394 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.436487 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.439829 (+ 3342us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.439976 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:44:53.440223 6403 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.440845 6405 heartbeater.cc:347] Connected to a master server at 127.30.196.61:35057
I20260928 16:44:53.440933 6405 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.441497 6405 heartbeater.cc:511] Master 127.30.196.61:35057 requested a full tablet report, sending...
I20260928 16:44:53.441601 6404 heartbeater.cc:347] Connected to a master server at 127.30.196.62:38891
I20260928 16:44:53.441682 6404 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:53.441838 6404 heartbeater.cc:511] Master 127.30.196.62:38891 requested a full tablet report, sending...
I20260928 16:44:53.442620 5158 ts_manager.cc:194] Re-registered known tserver with Master: 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:53.442667 5764 ts_manager.cc:194] Registered new tserver with Master: 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:53.441497 6403 heartbeater.cc:511] Master 127.30.196.60:38201 requested a full tablet report, sending...
I20260928 16:44:53.443338 5158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:55205
I20260928 16:44:53.444211 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.432384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.30.196.61:35057)
0928 16:44:53.433304 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.433308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.433712 (+ 404us) server_negotiation.cc:407] Connection header received
0928 16:44:53.433753 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.433756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.434089 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.434178 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.435277 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.436216 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.437469 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.437815 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.437918 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.440074 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.440078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.440080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.440105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.441901 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.442405 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.442409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.442410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.442498 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.442821 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.442825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.442827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.443035 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.443157 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.443915 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.444043 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":77,"threads_started":1}
I20260928 16:44:53.444397 5948 ts_manager.cc:194] Registered new tserver with Master: 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:53.450001 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.432312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.30.196.62:38891)
0928 16:44:53.432629 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.432633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.433036 (+ 403us) server_negotiation.cc:407] Connection header received
0928 16:44:53.433193 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.433197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.433256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.441589 (+ 8333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.441691 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.442401 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.443542 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.443900 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.444390 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.445623 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.445628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.445630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.445655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.447390 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.448687 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.448690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.448691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.448771 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.449077 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.449081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.449083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.449263 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.449409 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.449658 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.449790 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260928 16:44:53.452355 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.432817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.30.196.60:38201)
0928 16:44:53.437762 (+ 4945us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.437767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.438569 (+ 802us) server_negotiation.cc:407] Connection header received
0928 16:44:53.440529 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.440533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.440592 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.440675 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.445316 (+ 4641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.445969 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.447047 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.447279 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.447398 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.448706 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.448709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.448711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.448734 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.450402 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.450879 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.450884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.450888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.450983 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.451289 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.451292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.451294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.451593 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.451771 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.452078 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.452204 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890,"thread_start_us":98,"threads_started":1}
I20260928 16:44:53.453418 5158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:53.456681 5158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:53.459334 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: NON_VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: true } } }
I20260928 16:44:53.459486 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:53.462555 6421 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: attempting to promote NON_VOTER 763a5cae2d414b0dbf4f583edd06c602 to VOTER
I20260928 16:44:53.462885 6421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:53.464329 5774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 16:44:53.464846 5723 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:44:53.464979 5958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 LEARNER]: Refusing update from remote peer 023b4ece83dc4941b3bb5071f0574017: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 16:44:53.465378 5723 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:44:53.466694 5723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.467187 5774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.467411 5958 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.468308 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.468417 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:53.468684 6423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.468801 6423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:53.469141 6423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.469264 6423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:53.470276 5988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.470414 5988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:53.470348 6432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023b4ece83dc4941b3bb5071f0574017" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:53.470466 6432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:53.477952 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.472418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45073 (local address 127.0.0.1:46234)
0928 16:44:53.472532 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.472550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.472637 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.473909 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.473912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.473926 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.474035 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.474042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.475357 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.475364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.476217 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.476229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.477557 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.477560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.477646 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.477670 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.477733 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.477818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 16:44:53.478067 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.472660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.30.196.2:45073)
0928 16:44:53.473028 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.473040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.473068 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:44:53.473135 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.473140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.473406 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.473958 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.474171 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.475251 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.476346 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.477442 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.477561 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.477711 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.477768 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.477863 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
I20260928 16:44:53.478413 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.472691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41643 (local address 127.0.0.1:58402)
0928 16:44:53.473040 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.473053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.473145 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.474392 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.474395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.474412 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.474519 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.474529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.475770 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.475774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.476964 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.476974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.478011 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.478015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.478118 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.478144 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.478199 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.478274 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":52,"thread_start_us":134,"threads_started":1}
I20260928 16:44:53.478776 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.473049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45003 (local address 127.0.0.1:35268)
0928 16:44:53.473316 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.473329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.473408 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.474790 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.474793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.474810 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.474893 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.474899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.476238 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.476241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.477277 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.477288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.478391 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.478394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.478486 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.478509 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.478579 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.478657 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":46,"thread_start_us":93,"threads_started":1}
I20260928 16:44:53.478876 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.473602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.30.196.1:45003)
0928 16:44:53.474239 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.474254 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.474275 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:53.474343 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.474348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.474517 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.474656 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.475029 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.476126 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.477453 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.478270 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.478381 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.478565 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.478626 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.478720 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":85,"threads_started":1}
I20260928 16:44:53.479219 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.473598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.30.196.3:41643)
0928 16:44:53.474013 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.474026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.474048 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:53.474126 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.474129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.474261 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.474413 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.474615 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.475607 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.477093 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.477875 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.478670 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.478802 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.478961 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.479078 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
I20260928 16:44:53.481803 6336 tablet_service.cc:1467] Processing CreateTablet for tablet bebac0db58da487783f4fa34dc9024fd (DEFAULT_TABLE table=test-table [id=afcc0d18d2c449e095ab51686ef79ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:53.481808 6199 tablet_service.cc:1467] Processing CreateTablet for tablet bebac0db58da487783f4fa34dc9024fd (DEFAULT_TABLE table=test-table [id=afcc0d18d2c449e095ab51686ef79ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:53.481808 6062 tablet_service.cc:1467] Processing CreateTablet for tablet bebac0db58da487783f4fa34dc9024fd (DEFAULT_TABLE table=test-table [id=afcc0d18d2c449e095ab51686ef79ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:53.482837 6336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebac0db58da487783f4fa34dc9024fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:53.482844 6199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebac0db58da487783f4fa34dc9024fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:53.482843 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebac0db58da487783f4fa34dc9024fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:53.490203 6438 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Bootstrap starting.
I20260928 16:44:53.491420 6440 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Bootstrap starting.
I20260928 16:44:53.491576 6439 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Bootstrap starting.
I20260928 16:44:53.493520 6438 tablet_bootstrap.cc:654] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:53.493613 6439 tablet_bootstrap.cc:654] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:53.493613 6440 tablet_bootstrap.cc:654] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:53.494350 6438 log.cc:824] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:53.494458 6440 log.cc:824] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:53.494458 6439 log.cc:824] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:53.496469 6438 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: No bootstrap required, opened a new log
I20260928 16:44:53.496523 6440 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: No bootstrap required, opened a new log
I20260928 16:44:53.496526 6439 tablet_bootstrap.cc:492] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: No bootstrap required, opened a new log
I20260928 16:44:53.496706 6438 ts_tablet_manager.cc:1397] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:44:53.496707 6439 ts_tablet_manager.cc:1397] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:53.496707 6440 ts_tablet_manager.cc:1397] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:53.501473 6438 raft_consensus.cc:348] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.501760 6438 raft_consensus.cc:374] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:53.501816 6438 raft_consensus.cc:729] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b37bff0523e4c38bc0e69ba2627f45f, State: Initialized, Role: FOLLOWER
I20260928 16:44:53.501686 6440 raft_consensus.cc:348] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.501685 6439 raft_consensus.cc:348] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.501948 6440 raft_consensus.cc:374] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:53.501950 6439 raft_consensus.cc:374] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:53.502017 6439 raft_consensus.cc:729] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964e20401b50486da73d8a54751fb06e, State: Initialized, Role: FOLLOWER
I20260928 16:44:53.502017 6440 raft_consensus.cc:729] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5777e0b099453aab44c047d045e60b, State: Initialized, Role: FOLLOWER
I20260928 16:44:53.502379 6438 consensus_queue.cc:260] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.502453 6440 consensus_queue.cc:260] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.502452 6439 consensus_queue.cc:260] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.503494 6438 ts_tablet_manager.cc:1428] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:44:53.503507 6439 ts_tablet_manager.cc:1428] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:44:53.503499 6440 ts_tablet_manager.cc:1428] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:44:53.503803 6404 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
I20260928 16:44:53.504820 6129 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
I20260928 16:44:53.504992 6266 heartbeater.cc:503] Master 127.30.196.62:38891 was elected leader, sending a full tablet report...
W20260928 16:44:53.672176 6406 tablet.cc:2369] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:53.691186 6445 raft_consensus.cc:482] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:53.691565 6445 raft_consensus.cc:504] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.692865 6445 leader_election.cc:288] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073), 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:53.696933 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.693213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45073 (local address 127.30.196.1:57953)
0928 16:44:53.693547 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.693560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.693710 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.694012 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.694015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.694033 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.694139 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.694144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.694892 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.694896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.695802 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.695812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.696484 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.696488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.696589 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.696617 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.696703 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.696785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260928 16:44:53.696924 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.693394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:57953 (local address 127.30.196.2:45073)
0928 16:44:53.693597 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.693602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.693625 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:53.693839 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.693842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.693893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.693988 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.694242 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.694777 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.695927 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.696345 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.696488 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.696546 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.696721 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.696795 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 16:44:53.697118 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.693371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:45171 (local address 127.30.196.3:41643)
0928 16:44:53.693542 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.693548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.693946 (+ 398us) server_negotiation.cc:407] Connection header received
0928 16:44:53.694120 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.694122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.694174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.694260 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:53.694530 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.695030 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.696135 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.696560 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.696693 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.696744 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.696924 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.697000 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:53.697134 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.693213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41643 (local address 127.30.196.1:45171)
0928 16:44:53.693854 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:53.693871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:53.693999 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:53.694272 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:53.694275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:53.694291 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:53.694420 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.694425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.695149 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.695153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.696008 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:53.696018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.696688 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.696692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:53.696792 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.696819 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:44:53.696907 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:44:53.696990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":350,"thread_start_us":150,"threads_started":1}
I20260928 16:44:53.697791 6219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebac0db58da487783f4fa34dc9024fd" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964e20401b50486da73d8a54751fb06e" is_pre_election: true
I20260928 16:44:53.698133 6219 raft_consensus.cc:2454] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 0.
I20260928 16:44:53.698117 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebac0db58da487783f4fa34dc9024fd" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" is_pre_election: true
I20260928 16:44:53.698451 6357 raft_consensus.cc:2454] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 0.
I20260928 16:44:53.698683 6017 leader_election.cc:302] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5777e0b099453aab44c047d045e60b, 964e20401b50486da73d8a54751fb06e; no voters:
I20260928 16:44:53.698962 6445 raft_consensus.cc:2781] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:53.699055 6445 raft_consensus.cc:482] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:53.699112 6445 raft_consensus.cc:3037] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:53.700430 6445 raft_consensus.cc:504] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.700862 6445 leader_election.cc:288] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 election: Requested vote from peers 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073), 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643)
I20260928 16:44:53.701295 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebac0db58da487783f4fa34dc9024fd" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
I20260928 16:44:53.701304 6219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebac0db58da487783f4fa34dc9024fd" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964e20401b50486da73d8a54751fb06e"
I20260928 16:44:53.701411 6357 raft_consensus.cc:3037] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:53.701453 6219 raft_consensus.cc:3037] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:53.702574 6357 raft_consensus.cc:2454] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 1.
I20260928 16:44:53.702605 6219 raft_consensus.cc:2454] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 1.
I20260928 16:44:53.702888 6015 leader_election.cc:302] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b37bff0523e4c38bc0e69ba2627f45f, 8f5777e0b099453aab44c047d045e60b; no voters:
I20260928 16:44:53.703096 6445 raft_consensus.cc:2781] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:53.703310 6445 raft_consensus.cc:686] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 1 LEADER]: Becoming Leader. State: Replica: 8f5777e0b099453aab44c047d045e60b, State: Running, Role: LEADER
I20260928 16:44:53.703743 6445 consensus_queue.cc:237] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:53.708269 5156 catalog_manager.cc:5495] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f5777e0b099453aab44c047d045e60b (127.30.196.1). New cstate: current_term: 1 leader_uuid: "8f5777e0b099453aab44c047d045e60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } health_report { overall_health: HEALTHY } } }
W20260928 16:44:53.711325 6268 tablet.cc:2369] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:53.747997 6131 tablet.cc:2369] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:53.771834 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.768143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.30.196.1:45003)
0928 16:44:53.768272 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.768276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.768291 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:53.768454 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.768456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.768532 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.768625 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:53.768894 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.769417 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.770318 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.770540 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.770651 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.770913 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:53.771443 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:53.771545 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.771591 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.771645 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:44:53.777794 6357 raft_consensus.cc:1261] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Refusing update from remote peer 8f5777e0b099453aab44c047d045e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:53.777840 6219 raft_consensus.cc:1261] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Refusing update from remote peer 8f5777e0b099453aab44c047d045e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:53.778551 6445 consensus_queue.cc:1035] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:53.778817 6449 consensus_queue.cc:1035] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:53.786765 6452 mvcc.cc:205] Tried to move back new op lower bound from 7334354508906610688 to 7334354508613713920. Current Snapshot: MvccSnapshot[applied={T|T < 7334354508906610688}]
I20260928 16:44:53.792362 6453 mvcc.cc:205] Tried to move back new op lower bound from 7334354508906610688 to 7334354508613713920. Current Snapshot: MvccSnapshot[applied={T|T < 7334354508906610688}]
I20260928 16:44:53.797479 5156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:53.797704 5156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:53.803822 5156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 023b4ece83dc4941b3bb5071f0574017: Sending DeleteTablet for 3 replicas of tablet bebac0db58da487783f4fa34dc9024fd
I20260928 16:44:53.804636 6199 tablet_service.cc:1514] Processing DeleteTablet for tablet bebac0db58da487783f4fa34dc9024fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:53 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:46234
I20260928 16:44:53.804708 6336 tablet_service.cc:1514] Processing DeleteTablet for tablet bebac0db58da487783f4fa34dc9024fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:53 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:58402
I20260928 16:44:53.804858 6062 tablet_service.cc:1514] Processing DeleteTablet for tablet bebac0db58da487783f4fa34dc9024fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:53 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:35268
I20260928 16:44:53.805398 6464 tablet_replica.cc:330] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: stopping tablet replica
I20260928 16:44:53.805786 6464 raft_consensus.cc:2229] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:53.806198 6464 raft_consensus.cc:2258] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:53.807729 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5126
I20260928 16:44:53.808559 6464 ts_tablet_manager.cc:1905] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:53.810972 6464 ts_tablet_manager.cc:1918] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:53.811096 6464 log.cc:1196] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/bebac0db58da487783f4fa34dc9024fd
I20260928 16:44:53.811431 6464 ts_tablet_manager.cc:1939] T bebac0db58da487783f4fa34dc9024fd P 8f5777e0b099453aab44c047d045e60b: Deleting consensus metadata
I20260928 16:44:53.812589 6462 tablet_replica.cc:330] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: stopping tablet replica
I20260928 16:44:53.812990 6462 raft_consensus.cc:2229] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:53.813364 6462 raft_consensus.cc:2258] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:53.813928 6463 tablet_replica.cc:330] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: stopping tablet replica
I20260928 16:44:53.814275 6463 raft_consensus.cc:2229] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:53.814612 6463 raft_consensus.cc:2258] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:53.815054 6462 ts_tablet_manager.cc:1905] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:53.816243 6463 ts_tablet_manager.cc:1905] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:44:53.817634 6018 connection.cc:535] server connection from 127.0.0.1:35268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:44:53.818382 6462 ts_tablet_manager.cc:1918] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:53.818518 6462 log.cc:1196] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/bebac0db58da487783f4fa34dc9024fd
I20260928 16:44:53.818908 6462 ts_tablet_manager.cc:1939] T bebac0db58da487783f4fa34dc9024fd P 964e20401b50486da73d8a54751fb06e: Deleting consensus metadata
W20260928 16:44:53.820402 6153 connection.cc:441] server connection from 127.0.0.1:46234 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46234 (request call id 1) could send its response
I20260928 16:44:53.821429 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.806196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.30.196.61:35057)
0928 16:44:53.806310 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.806315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.806328 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:53.806367 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.806370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.806417 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.806496 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.812413 (+ 5917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.813267 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.814580 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.814912 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.815008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.817166 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.817173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.817179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.817215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.819173 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.819958 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.819962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.819964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.820075 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.820403 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.820407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.820410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.820671 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.820771 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.821131 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.821267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:44:53.831231 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.806201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.30.196.60:38201)
0928 16:44:53.806304 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.806308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.821490 (+ 15182us) server_negotiation.cc:407] Connection header received
0928 16:44:53.821694 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.821698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.821753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.821810 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.822595 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.823388 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.824526 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.824876 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.825002 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.826672 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.826676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.826679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.826708 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.828496 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.829006 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.829009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.829011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.829104 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.829768 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.829774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.829778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.830033 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.830169 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.830896 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.831023 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 16:44:53.832381 6463 ts_tablet_manager.cc:1918] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:53.832532 6463 log.cc:1196] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/bebac0db58da487783f4fa34dc9024fd
I20260928 16:44:53.832934 6463 ts_tablet_manager.cc:1939] T bebac0db58da487783f4fa34dc9024fd P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting consensus metadata
W20260928 16:44:53.834417 6291 connection.cc:441] server connection from 127.0.0.1:58402 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58402 (request call id 1) could send its response
I20260928 16:44:53.834955 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.823000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.30.196.61:35057)
0928 16:44:53.823727 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.823731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.823744 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:53.823777 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.823779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.823820 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.823894 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.826088 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.826870 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.828222 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.828603 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.828684 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.830787 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.830793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.830796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.830830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.833127 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.833583 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.833586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.833588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.833677 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.833957 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.833961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.833963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.834151 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.834287 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.834609 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.834838 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629}
I20260928 16:44:53.835089 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:53.824085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.30.196.60:38201)
0928 16:44:53.824216 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:53.824221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:53.824577 (+ 356us) server_negotiation.cc:407] Connection header received
0928 16:44:53.824820 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:53.824824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:53.824874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:53.824963 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:53.826595 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.827396 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.828703 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:53.829079 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:53.829155 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:53.830895 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:53.830898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:53.830901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:53.830928 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:53.832859 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.833414 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.833417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.833419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.833510 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:53.833840 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:53.833843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:53.833845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:53.834063 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:53.834171 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:53.834632 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 16:44:53.834931 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:44:54.795361 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:54.794839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:47269)
0928 16:44:54.795133 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:54.795237 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260928 16:44:54.795405 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:54.794573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:60755)
0928 16:44:54.794861 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:54.795292 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
W20260928 16:44:54.795917 6266 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:38891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
W20260928 16:44:54.796036 6129 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:38891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:54.796496 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:54.796021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:43083)
0928 16:44:54.796292 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:54.796383 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
W20260928 16:44:54.796721 6404 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:38891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:55.303578 6476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 023b4ece83dc4941b3bb5071f0574017)
I20260928 16:44:55.303890 6476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:55.303958 6477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 023b4ece83dc4941b3bb5071f0574017)
I20260928 16:44:55.304139 6477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:55.305054 6477 leader_election.cc:288] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61:35057)
I20260928 16:44:55.305199 6476 leader_election.cc:288] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201)
I20260928 16:44:55.305485 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.304863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32940)
0928 16:44:55.305258 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.305366 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":56,"threads_started":1}
I20260928 16:44:55.305723 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.305287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32952)
0928 16:44:55.305545 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.305624 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
W20260928 16:44:55.305923 5935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
W20260928 16:44:55.305941 5751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:55.306871 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.306583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32956)
0928 16:44:55.306729 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.306778 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 16:44:55.307008 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.306729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32964)
0928 16:44:55.306870 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.306922 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 16:44:55.307027 5935 leader_election.cc:334] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
W20260928 16:44:55.307114 5751 leader_election.cc:334] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:55.309813 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.304863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:35057 (local address 127.0.0.1:35702)
0928 16:44:55.305703 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.305718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:55.305948 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:55.306568 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:55.306572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:55.306593 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:55.306723 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:55.306730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.307896 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.307898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:55.308754 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:55.308763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.309421 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.309424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:55.309484 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:55.309499 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:44:55.309584 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:44:55.309668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":65,"threads_started":1}
I20260928 16:44:55.309818 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.305691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:38201 (local address 127.0.0.1:60412)
0928 16:44:55.305855 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.305869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:55.305980 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:55.306702 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:55.306705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:55.306724 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:55.306825 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:55.306832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.307783 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.307786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:55.308631 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:55.308641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.309327 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.309333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:55.309489 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:55.309509 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:44:55.309614 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:44:55.309697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":62,"thread_start_us":66,"threads_started":1}
I20260928 16:44:55.309814 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.306179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.30.196.60:38201)
0928 16:44:55.306413 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:55.306455 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:55.306470 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:55.306520 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:55.306526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:55.306574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:55.306951 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:55.307060 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.307637 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.308761 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.309210 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.309293 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:55.309369 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:55.309637 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 16:44:55.309693 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260928 16:44:55.309820 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.304966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.30.196.61:35057)
0928 16:44:55.305343 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:55.305348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:55.305862 (+ 514us) server_negotiation.cc:407] Connection header received
0928 16:44:55.306268 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:55.306272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:55.306333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:55.306433 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:55.306850 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.307794 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.308869 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:55.309298 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:55.309397 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:55.309502 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:55.309588 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:44:55.309685 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":261,"thread_start_us":76,"threads_started":1}
I20260928 16:44:55.310314 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5882f3a266364dfa9ace75c6dad1f993" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "763a5cae2d414b0dbf4f583edd06c602" is_pre_election: true
I20260928 16:44:55.310541 5958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5882f3a266364dfa9ace75c6dad1f993 in term 1.
I20260928 16:44:55.310681 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "763a5cae2d414b0dbf4f583edd06c602" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5882f3a266364dfa9ace75c6dad1f993" is_pre_election: true
I20260928 16:44:55.310845 5774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763a5cae2d414b0dbf4f583edd06c602 in term 1.
I20260928 16:44:55.310971 5751 leader_election.cc:302] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5882f3a266364dfa9ace75c6dad1f993, 763a5cae2d414b0dbf4f583edd06c602; no voters: 023b4ece83dc4941b3bb5071f0574017
I20260928 16:44:55.311151 5934 leader_election.cc:302] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5882f3a266364dfa9ace75c6dad1f993, 763a5cae2d414b0dbf4f583edd06c602; no voters: 023b4ece83dc4941b3bb5071f0574017
I20260928 16:44:55.311200 6476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:44:55.311249 6476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 023b4ece83dc4941b3bb5071f0574017)
I20260928 16:44:55.311286 6476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:44:55.311352 6477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:44:55.311405 6477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 023b4ece83dc4941b3bb5071f0574017)
I20260928 16:44:55.311455 6477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:44:55.312444 6476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:55.312486 6477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:55.312870 6476 leader_election.cc:288] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 election: Requested vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201)
I20260928 16:44:55.312884 6477 leader_election.cc:288] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 election: Requested vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 5882f3a266364dfa9ace75c6dad1f993 (127.30.196.61:35057)
I20260928 16:44:55.313215 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "763a5cae2d414b0dbf4f583edd06c602" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5882f3a266364dfa9ace75c6dad1f993"
I20260928 16:44:55.313306 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.313000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32968)
0928 16:44:55.313138 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.313230 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:44:55.313381 5774 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 763a5cae2d414b0dbf4f583edd06c602 in current term 2: Already voted for candidate 5882f3a266364dfa9ace75c6dad1f993 in this term.
I20260928 16:44:55.313488 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.313248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32972)
0928 16:44:55.313372 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.313426 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:44:55.313740 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5882f3a266364dfa9ace75c6dad1f993" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "763a5cae2d414b0dbf4f583edd06c602"
I20260928 16:44:55.313923 5958 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5882f3a266364dfa9ace75c6dad1f993 in current term 2: Already voted for candidate 763a5cae2d414b0dbf4f583edd06c602 in this term.
I20260928 16:44:55.314189 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.313942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32978)
0928 16:44:55.314054 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.314122 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:44:55.314302 5935 leader_election.cc:334] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:55.314369 5935 leader_election.cc:302] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 763a5cae2d414b0dbf4f583edd06c602; no voters: 023b4ece83dc4941b3bb5071f0574017, 5882f3a266364dfa9ace75c6dad1f993
I20260928 16:44:55.314502 6477 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:44:55.314713 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.314503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:32988)
0928 16:44:55.314607 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.314654 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:44:55.314814 5751 leader_election.cc:334] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:55.314877 5751 leader_election.cc:302] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5882f3a266364dfa9ace75c6dad1f993; no voters: 023b4ece83dc4941b3bb5071f0574017, 763a5cae2d414b0dbf4f583edd06c602
I20260928 16:44:55.315011 6476 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:44:55.797505 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.796705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:54975)
0928 16:44:55.797303 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.797379 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":73,"threads_started":1}
I20260928 16:44:55.798125 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.796622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:60833)
0928 16:44:55.797929 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.797997 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":109,"threads_started":1}
I20260928 16:44:55.798131 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:55.797268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:49437)
0928 16:44:55.797870 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:55.798039 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":77,"threads_started":1}
I20260928 16:44:56.798915 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:56.798370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:38519)
0928 16:44:56.798717 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:56.798789 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":83,"threads_started":1}
I20260928 16:44:56.799372 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:56.798948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:40677)
0928 16:44:56.799214 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:56.799279 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
I20260928 16:44:56.799445 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:56.798998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:54695)
0928 16:44:56.799279 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:56.799344 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":60,"threads_started":1}
I20260928 16:44:57.152509 6492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:57.152654 6492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:57.153203 6492 leader_election.cc:288] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201)
I20260928 16:44:57.153954 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.153562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:33254)
0928 16:44:57.153781 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.153853 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260928 16:44:57.154206 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5882f3a266364dfa9ace75c6dad1f993" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "763a5cae2d414b0dbf4f583edd06c602" is_pre_election: true
I20260928 16:44:57.154376 5958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5882f3a266364dfa9ace75c6dad1f993 in term 2.
I20260928 16:44:57.154790 5751 leader_election.cc:302] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5882f3a266364dfa9ace75c6dad1f993, 763a5cae2d414b0dbf4f583edd06c602; no voters:
I20260928 16:44:57.154882 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.154630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:33264)
0928 16:44:57.154743 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.154795 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:44:57.154960 5751 leader_election.cc:334] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:57.155017 6492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 16:44:57.155100 6492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:57.155154 6492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:44:57.156312 6492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:57.156706 6492 leader_election.cc:288] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 election: Requested vote from peers 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891), 763a5cae2d414b0dbf4f583edd06c602 (127.30.196.60:38201)
I20260928 16:44:57.157083 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.156844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:33280)
0928 16:44:57.156949 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.157007 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 16:44:57.157509 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5882f3a266364dfa9ace75c6dad1f993" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "763a5cae2d414b0dbf4f583edd06c602"
I20260928 16:44:57.157649 5958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:44:57.157845 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.157592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:33294)
0928 16:44:57.157747 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.157794 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 16:44:57.158003 5751 leader_election.cc:334] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
I20260928 16:44:57.158808 5958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5882f3a266364dfa9ace75c6dad1f993 in term 3.
I20260928 16:44:57.159128 5751 leader_election.cc:302] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5882f3a266364dfa9ace75c6dad1f993, 763a5cae2d414b0dbf4f583edd06c602; no voters: 023b4ece83dc4941b3bb5071f0574017
I20260928 16:44:57.159281 6492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:44:57.159507 6492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [term 3 LEADER]: Becoming Leader. State: Replica: 5882f3a266364dfa9ace75c6dad1f993, State: Running, Role: LEADER
I20260928 16:44:57.159817 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } }
I20260928 16:44:57.160832 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5882f3a266364dfa9ace75c6dad1f993. Latest consensus state: current_term: 3 leader_uuid: "5882f3a266364dfa9ace75c6dad1f993" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:57.160962 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:57.161340 6497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:57.163832 6497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:57.164294 6497 catalog_manager.cc:1259] Loaded cluster ID: 29caaffa718b4eb89e95b57edc2b8257
I20260928 16:44:57.164362 6497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:57.165017 6497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:57.165413 6497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Loaded TSK: 0
I20260928 16:44:57.166090 6497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:57.225238 5764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:57.228577 5764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:57.231741 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.231380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:33312)
0928 16:44:57.231545 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.231604 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:44:57.231772 5958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [term 3 FOLLOWER]: Refusing update from remote peer 5882f3a266364dfa9ace75c6dad1f993: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 16:44:57.232261 6492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 16:44:57.232650 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.232398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.0.0.1:42598)
0928 16:44:57.232513 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.232564 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:44:57.232795 5751 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 -> Peer 023b4ece83dc4941b3bb5071f0574017 (127.30.196.62:38891): Couldn't send request to peer 023b4ece83dc4941b3bb5071f0574017. Status: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:57.233827 6499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5882f3a266364dfa9ace75c6dad1f993. Latest consensus state: current_term: 3 leader_uuid: "5882f3a266364dfa9ace75c6dad1f993" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:57.233968 6499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:57.235152 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5882f3a266364dfa9ace75c6dad1f993" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:57.235284 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:57.235594 6500 mvcc.cc:205] Tried to move back new op lower bound from 7334354523055923200 to 7334354522768990208. Current Snapshot: MvccSnapshot[applied={T|T < 7334354523055923200}]
I20260928 16:44:57.236784 6499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5882f3a266364dfa9ace75c6dad1f993" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "023b4ece83dc4941b3bb5071f0574017" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38891 } } peers { permanent_uuid: "5882f3a266364dfa9ace75c6dad1f993" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 35057 } attrs { promote: false } } peers { permanent_uuid: "763a5cae2d414b0dbf4f583edd06c602" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 38201 } attrs { promote: false } } }
I20260928 16:44:57.236899 6499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5cae2d414b0dbf4f583edd06c602 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:57.246655 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.241803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45073 (local address 127.0.0.1:59752)
0928 16:44:57.242316 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.242332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:57.242421 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:57.242898 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:57.242904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:57.242923 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:57.243137 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.243146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.244035 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.244039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.245236 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.245247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.246178 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.246307 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.246349 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:44:57.246398 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 16:44:57.246496 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":26,"thread_start_us":132,"threads_started":1}
I20260928 16:44:57.246690 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.241583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41643 (local address 127.0.0.1:47214)
0928 16:44:57.241703 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.241718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:57.241809 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:57.243036 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:57.243040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:57.243067 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:57.243244 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.243253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.244294 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.244298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.245464 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.245471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.246299 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.246410 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.246429 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:44:57.246499 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:44:57.246579 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":111,"spinlock_wait_cycles":768}
I20260928 16:44:57.246755 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.242057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.30.196.3:41643)
0928 16:44:57.242790 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:57.242795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:57.242813 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:57.242866 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:57.242868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:57.242912 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:57.243021 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:57.243370 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.244191 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.245583 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.246203 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246301 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.246375 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:57.246537 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:44:57.246613 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260928 16:44:57.246790 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.242431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.30.196.1:45003)
0928 16:44:57.242695 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:57.242699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:57.242713 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:57.242766 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:57.242769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:57.242819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:57.242917 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:57.243291 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.244161 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.245544 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.245991 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246118 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.246191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:57.246564 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:44:57.246653 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260928 16:44:57.247014 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.242083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45003 (local address 127.0.0.1:45426)
0928 16:44:57.242446 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.242457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:57.242532 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:57.243025 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:57.243028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:57.243042 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:57.243166 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.243175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.244298 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.244301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.245415 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:57.245425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.246116 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:57.246231 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.246257 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:44:57.246331 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:44:57.246406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":67,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260928 16:44:57.247210 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.242127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.30.196.2:45073)
0928 16:44:57.242638 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:57.242644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:57.242659 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:57.242718 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:57.242723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:57.242778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:57.242870 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:57.243268 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.243923 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.245368 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.246037 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.246956 (+ 919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.247008 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:57.247049 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 16:44:57.247111 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":81,"threads_started":1}
I20260928 16:44:57.247784 6199 tablet_service.cc:1467] Processing CreateTablet for tablet f63ee836c7224c4d97fc14827c4faa11 (DEFAULT_TABLE table=test-table [id=7e261768dbd94ebc9bb786a6fe228813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:57.247784 6336 tablet_service.cc:1467] Processing CreateTablet for tablet f63ee836c7224c4d97fc14827c4faa11 (DEFAULT_TABLE table=test-table [id=7e261768dbd94ebc9bb786a6fe228813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:57.248085 6336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ee836c7224c4d97fc14827c4faa11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:57.248086 6199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ee836c7224c4d97fc14827c4faa11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:57.248104 6062 tablet_service.cc:1467] Processing CreateTablet for tablet f63ee836c7224c4d97fc14827c4faa11 (DEFAULT_TABLE table=test-table [id=7e261768dbd94ebc9bb786a6fe228813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:57.248401 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ee836c7224c4d97fc14827c4faa11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:57.250474 6510 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: Bootstrap starting.
I20260928 16:44:57.251369 6511 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: Bootstrap starting.
I20260928 16:44:57.251638 6510 tablet_bootstrap.cc:654] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:57.252313 6511 tablet_bootstrap.cc:654] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:57.253010 6512 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Bootstrap starting.
I20260928 16:44:57.253324 6510 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: No bootstrap required, opened a new log
I20260928 16:44:57.253424 6510 ts_tablet_manager.cc:1397] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:44:57.253639 6511 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: No bootstrap required, opened a new log
I20260928 16:44:57.253727 6511 ts_tablet_manager.cc:1397] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:57.253902 6510 raft_consensus.cc:348] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.253985 6512 tablet_bootstrap.cc:654] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:57.254031 6510 raft_consensus.cc:374] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:57.254069 6510 raft_consensus.cc:729] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5777e0b099453aab44c047d045e60b, State: Initialized, Role: FOLLOWER
I20260928 16:44:57.254195 6511 raft_consensus.cc:348] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.254204 6510 consensus_queue.cc:260] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.254300 6511 raft_consensus.cc:374] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:57.254333 6511 raft_consensus.cc:729] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964e20401b50486da73d8a54751fb06e, State: Initialized, Role: FOLLOWER
I20260928 16:44:57.254477 6511 consensus_queue.cc:260] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.254608 6510 ts_tablet_manager.cc:1428] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 16:44:57.254817 6511 ts_tablet_manager.cc:1428] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:57.255126 6512 tablet_bootstrap.cc:492] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: No bootstrap required, opened a new log
I20260928 16:44:57.255206 6512 ts_tablet_manager.cc:1397] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:57.255734 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.255252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:35209)
0928 16:44:57.255558 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.255618 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260928 16:44:57.255658 6512 raft_consensus.cc:348] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.255789 6512 raft_consensus.cc:374] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:57.255822 6512 raft_consensus.cc:729] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b37bff0523e4c38bc0e69ba2627f45f, State: Initialized, Role: FOLLOWER
I20260928 16:44:57.255950 6512 consensus_queue.cc:260] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.256142 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.255726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:44915)
0928 16:44:57.256018 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.256073 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260928 16:44:57.256312 6512 ts_tablet_manager.cc:1428] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:44:57.257854 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.257609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:36627)
0928 16:44:57.257733 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.257797 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:44:57.259317 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.259073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:49177)
0928 16:44:57.259209 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.259256 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:44:57.260064 6405 heartbeater.cc:503] Master 127.30.196.61:35057 was elected leader, sending a full tablet report...
I20260928 16:44:57.260003 6130 heartbeater.cc:503] Master 127.30.196.61:35057 was elected leader, sending a full tablet report...
I20260928 16:44:57.260617 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.260217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:54249)
0928 16:44:57.260492 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.260546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260928 16:44:57.261324 6267 heartbeater.cc:503] Master 127.30.196.61:35057 was elected leader, sending a full tablet report...
I20260928 16:44:57.261934 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.261521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:35125)
0928 16:44:57.261811 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.261864 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260928 16:44:57.523829 6515 raft_consensus.cc:482] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:57.524070 6515 raft_consensus.cc:504] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.524624 6515 leader_election.cc:288] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643), 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:57.525280 6219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ee836c7224c4d97fc14827c4faa11" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964e20401b50486da73d8a54751fb06e" is_pre_election: true
I20260928 16:44:57.525280 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ee836c7224c4d97fc14827c4faa11" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" is_pre_election: true
I20260928 16:44:57.525441 6219 raft_consensus.cc:2454] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 0.
I20260928 16:44:57.525441 6357 raft_consensus.cc:2454] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 0.
I20260928 16:44:57.525837 6017 leader_election.cc:302] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5777e0b099453aab44c047d045e60b, 964e20401b50486da73d8a54751fb06e; no voters:
I20260928 16:44:57.526131 6515 raft_consensus.cc:2781] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:57.526194 6515 raft_consensus.cc:482] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:57.526255 6515 raft_consensus.cc:3037] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:57.528820 6515 raft_consensus.cc:504] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.529248 6515 leader_election.cc:288] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 election: Requested vote from peers 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643), 964e20401b50486da73d8a54751fb06e (127.30.196.2:45073)
I20260928 16:44:57.529808 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ee836c7224c4d97fc14827c4faa11" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37bff0523e4c38bc0e69ba2627f45f"
I20260928 16:44:57.529812 6219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ee836c7224c4d97fc14827c4faa11" candidate_uuid: "8f5777e0b099453aab44c047d045e60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964e20401b50486da73d8a54751fb06e"
I20260928 16:44:57.529956 6357 raft_consensus.cc:3037] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:57.530033 6219 raft_consensus.cc:3037] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:57.531183 6219 raft_consensus.cc:2454] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 1.
I20260928 16:44:57.531177 6357 raft_consensus.cc:2454] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5777e0b099453aab44c047d045e60b in term 1.
I20260928 16:44:57.531517 6017 leader_election.cc:302] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5777e0b099453aab44c047d045e60b, 964e20401b50486da73d8a54751fb06e; no voters:
I20260928 16:44:57.531764 6515 raft_consensus.cc:2781] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:57.532054 6515 raft_consensus.cc:686] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 1 LEADER]: Becoming Leader. State: Replica: 8f5777e0b099453aab44c047d045e60b, State: Running, Role: LEADER
I20260928 16:44:57.532233 6515 consensus_queue.cc:237] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } }
I20260928 16:44:57.533071 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.532751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:44989)
0928 16:44:57.532913 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.532981 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:44:57.534072 5763 catalog_manager.cc:5495] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f5777e0b099453aab44c047d045e60b (127.30.196.1). New cstate: current_term: 1 leader_uuid: "8f5777e0b099453aab44c047d045e60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f5777e0b099453aab44c047d045e60b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45003 } health_report { overall_health: HEALTHY } } }
I20260928 16:44:57.615267 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.612266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.30.196.1:45003)
0928 16:44:57.612396 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:57.612400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:57.612418 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:57.612562 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:57.612565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:57.612613 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:57.612681 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:57.612986 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.613544 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.614384 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:57.614600 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:57.614681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:57.614907 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:57.614976 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:57.615034 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:57.615108 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:44:57.615155 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:44:57.617573 6219 raft_consensus.cc:1261] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Refusing update from remote peer 8f5777e0b099453aab44c047d045e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:57.617570 6357 raft_consensus.cc:1261] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Refusing update from remote peer 8f5777e0b099453aab44c047d045e60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:57.618086 6515 consensus_queue.cc:1035] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "964e20401b50486da73d8a54751fb06e" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:57.618331 6519 consensus_queue.cc:1035] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b37bff0523e4c38bc0e69ba2627f45f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:57.619957 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.619588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:49695)
0928 16:44:57.619773 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.619849 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 16:44:57.620520 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.620232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:42895)
0928 16:44:57.620387 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.620442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 16:44:57.623407 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.623110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:35831)
0928 16:44:57.623279 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.623334 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 16:44:57.624039 6524 mvcc.cc:205] Tried to move back new op lower bound from 7334354524637274112 to 7334354524293103616. Current Snapshot: MvccSnapshot[applied={T|T < 7334354524637274112}]
I20260928 16:44:57.626001 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.625660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.1:35231)
0928 16:44:57.625846 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.625903 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 16:44:57.627697 5762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:57.627880 5762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:57.628187 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.627877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.3:56537)
0928 16:44:57.628052 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.628109 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 16:44:57.629215 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.628916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38891 (local address 127.30.196.2:58373)
0928 16:44:57.629085 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.629144 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 16:44:57.632557 5762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5882f3a266364dfa9ace75c6dad1f993: Sending DeleteTablet for 3 replicas of tablet f63ee836c7224c4d97fc14827c4faa11
I20260928 16:44:57.633172 6336 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ee836c7224c4d97fc14827c4faa11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:57 UTC) from {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:47214
I20260928 16:44:57.633322 6199 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ee836c7224c4d97fc14827c4faa11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:57 UTC) from {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:59752
I20260928 16:44:57.633551 6062 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ee836c7224c4d97fc14827c4faa11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:57 UTC) from {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:45426
I20260928 16:44:57.633632 6198 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ee836c7224c4d97fc14827c4faa11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:57 UTC) from {username='oryx', principal='oryx/127.30.196.61@KRBTEST.COM'} at 127.0.0.1:59752
I20260928 16:44:57.633886 6534 tablet_replica.cc:330] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b: stopping tablet replica
I20260928 16:44:57.633919 6533 tablet_replica.cc:330] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: stopping tablet replica
I20260928 16:44:57.634051 6532 tablet_replica.cc:330] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: stopping tablet replica
I20260928 16:44:57.634043 6533 raft_consensus.cc:2229] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:57.634159 6532 raft_consensus.cc:2229] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:57.634091 6534 raft_consensus.cc:2229] T f63ee836c7224c4d97fc14827c4faa11 P 8f5777e0b099453aab44c047d045e60b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:57.634363 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5909
I20260928 16:44:57.634289 6532 raft_consensus.cc:2258] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:57.636824 6532 ts_tablet_manager.cc:1905] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:57.639453 6532 ts_tablet_manager.cc:1918] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:57.639539 6532 log.cc:1196] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/f63ee836c7224c4d97fc14827c4faa11
I20260928 16:44:57.639874 6532 ts_tablet_manager.cc:1939] T f63ee836c7224c4d97fc14827c4faa11 P 5b37bff0523e4c38bc0e69ba2627f45f: Deleting consensus metadata
I20260928 16:44:57.634202 6533 raft_consensus.cc:2258] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:57.640967 5748 catalog_manager.cc:4841] TS 5b37bff0523e4c38bc0e69ba2627f45f (127.30.196.3:41643): tablet f63ee836c7224c4d97fc14827c4faa11 (table test-table [id=7e261768dbd94ebc9bb786a6fe228813]) successfully deleted
I20260928 16:44:57.641211 6533 ts_tablet_manager.cc:1905] T f63ee836c7224c4d97fc14827c4faa11 P 964e20401b50486da73d8a54751fb06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:57.643321 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6136
I20260928 16:44:57.644372 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.643416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45003 (local address 127.0.0.1:45442)
0928 16:44:57.644186 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.644258 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":648}
W20260928 16:44:57.644570 5748 catalog_manager.cc:4584] TS 8f5777e0b099453aab44c047d045e60b (127.30.196.1:45003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f63ee836c7224c4d97fc14827c4faa11: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45003: connect: Connection refused (error 111)
I20260928 16:44:57.651151 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6272
I20260928 16:44:57.652158 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:57.651281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45073 (local address 127.0.0.1:59756)
0928 16:44:57.651972 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:57.652045 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561}
I20260928 16:44:57.657711 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5626
I20260928 16:44:57.664669 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 5730
2026-09-28T16:44:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7678 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:57 dist-test-slave-5rg1 krb5kdc[6537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:57 dist-test-slave-5rg1 krb5kdc[6537](info): set up 2 sockets
Sep 28 16:44:57 dist-test-slave-5rg1 krb5kdc[6537](info): commencing operation
krb5kdc: starting...
W20260928 16:44:58.720541 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:44:58 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab.
Entry for principal oryx/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab.
I20260928 16:44:58.851909 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45841
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab
--principal=oryx/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:45841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:58.964816 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:58.965102 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:58.965142 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:58.969213 6550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:58.969266 6550 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:44:58.969285 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:58.969302 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:58.969318 6550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:58.969333 6550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:58.974527 6550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:45841
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.62.keytab
--principal=oryx/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:58.975558 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:58.976616 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:58.985957 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:58.986028 6550 server_base.cc:1034] running on GCE node
W20260928 16:44:58.986099 6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:58.985966 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:58.986766 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:58.988195 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:58.989383 6550 hybrid_clock.cc:648] HybridClock initialized: now 1790613898989343 us; error 66 us; skew 500 ppm
Sep 28 16:44:58 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613898, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:58.992270 6550 init.cc:377] Logged in from keytab as oryx/127.30.196.62@KRBTEST.COM (short username oryx)
I20260928 16:44:58.993242 6550 webserver.cc:460] Webserver started at http://127.30.196.62:40735/ using document root <none> and password file <none>
I20260928 16:44:58.993844 6550 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:58.993937 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:58.994174 6550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:58.996454 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "da5895f8e74c49929dec6285976bbd54"
format_stamp: "Formatted at 2026-09-28 16:44:58 on dist-test-slave-5rg1"
server_key: "c6382220c31a5808775ef9138364390d"
server_key_iv: "d0c0ee346e12e6aadb891ad1c745a2ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:58.997123 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "da5895f8e74c49929dec6285976bbd54"
format_stamp: "Formatted at 2026-09-28 16:44:58 on dist-test-slave-5rg1"
server_key: "c6382220c31a5808775ef9138364390d"
server_key_iv: "d0c0ee346e12e6aadb891ad1c745a2ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.001160 6550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:59.003857 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.005206 6550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:59.005357 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "da5895f8e74c49929dec6285976bbd54"
format_stamp: "Formatted at 2026-09-28 16:44:58 on dist-test-slave-5rg1"
server_key: "c6382220c31a5808775ef9138364390d"
server_key_iv: "d0c0ee346e12e6aadb891ad1c745a2ee"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.005505 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:59.018898 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:59.022161 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:59.022383 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:59.031010 6550 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:45841
I20260928 16:44:59.031111 6628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:45841 every 8 connection(s)
I20260928 16:44:59.032124 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:59.035009 6629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:59.038826 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6550
I20260928 16:44:59.038944 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:44:59.039297 31504 external_mini_cluster.cc:1428] Setting key ec12080ae93072225d74d339a94e1327
I20260928 16:44:59.041142 6629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Bootstrap starting.
I20260928 16:44:59.043653 6629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:59.044456 6629 log.cc:824] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:59.046324 6629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: No bootstrap required, opened a new log
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:59.050184 6629 raft_consensus.cc:348] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } }
I20260928 16:44:59.050441 6629 raft_consensus.cc:374] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:59.050525 6629 raft_consensus.cc:729] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5895f8e74c49929dec6285976bbd54, State: Initialized, Role: FOLLOWER
I20260928 16:44:59.051015 6629 consensus_queue.cc:260] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } }
I20260928 16:44:59.051179 6629 raft_consensus.cc:388] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:59.051267 6629 raft_consensus.cc:482] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:59.051371 6629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:59.052366 6629 raft_consensus.cc:504] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } }
I20260928 16:44:59.052757 6629 leader_election.cc:302] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da5895f8e74c49929dec6285976bbd54; no voters:
I20260928 16:44:59.053140 6629 leader_election.cc:288] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:59.053236 6634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:59.053444 6634 raft_consensus.cc:686] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [term 1 LEADER]: Becoming Leader. State: Replica: da5895f8e74c49929dec6285976bbd54, State: Running, Role: LEADER
I20260928 16:44:59.053761 6634 consensus_queue.cc:237] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } }
I20260928 16:44:59.053848 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.040683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.30.196.62:45841)
0928 16:44:59.041091 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.041107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.041142 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:44:59.041826 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.041842 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.042170 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.042437 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.043326 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.044505 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.045401 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.045877 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.046062 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.048753 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.048758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.048772 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.048799 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.050603 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.051047 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.051055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.051065 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.051215 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.051538 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.051541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.051543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.052008 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.052225 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.052475 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.052781 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260928 16:44:59.054483 6629 sys_catalog.cc:564] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:59.055439 6635 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da5895f8e74c49929dec6285976bbd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } } }
I20260928 16:44:59.055562 6635 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:59.056989 6636 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da5895f8e74c49929dec6285976bbd54. Latest consensus state: current_term: 1 leader_uuid: "da5895f8e74c49929dec6285976bbd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da5895f8e74c49929dec6285976bbd54" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45841 } } }
I20260928 16:44:59.057396 6636 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54 [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:59.059459 6650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:59.059561 6650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:59.059717 6643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:59.060600 6643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:59.066340 6643 catalog_manager.cc:1347] Generated new cluster ID: 7e0e566f17f24307b316c15bf1e8b6a0
I20260928 16:44:59.066412 6643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:59.074965 6643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:59.076272 6643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:59.083527 6643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Generated new TSK 0
I20260928 16:44:59.084362 6643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
I20260928 16:44:59.120468 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45841
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:59.237957 6655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:59.238255 6655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:59.238303 6655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:59.242554 6655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:59.242646 6655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:59.242689 6655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:59.242724 6655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:44:59.248816 6655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:59.250046 6655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:59.251128 6655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:59.261848 6660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:59.261871 6661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:59.262120 6655 server_base.cc:1034] running on GCE node
W20260928 16:44:59.261876 6663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:59.262735 6655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:59.263439 6655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:59.264638 6655 hybrid_clock.cc:648] HybridClock initialized: now 1790613899264641 us; error 55 us; skew 500 ppm
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:59.267532 6655 init.cc:377] Logged in from keytab as oryx/127.30.196.1@KRBTEST.COM (short username oryx)
I20260928 16:44:59.268541 6655 webserver.cc:460] Webserver started at http://127.30.196.1:34569/ using document root <none> and password file <none>
I20260928 16:44:59.269219 6655 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:59.269321 6655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:59.269560 6655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:59.271783 6655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "6d78a091c5a3a3a727d0f2ee18680cf8"
server_key_iv: "552776c584588def580c00bf19c4fc77"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.272377 6655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "6d78a091c5a3a3a727d0f2ee18680cf8"
server_key_iv: "552776c584588def580c00bf19c4fc77"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.276192 6655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:59.278932 6670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.280448 6655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:59.280613 6655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "6d78a091c5a3a3a727d0f2ee18680cf8"
server_key_iv: "552776c584588def580c00bf19c4fc77"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.280761 6655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:59.294516 6655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:59.298192 6655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:59.298424 6655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:59.300017 6655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:59.300098 6655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.300179 6655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:59.300232 6655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:59.312179 6655 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:40663
I20260928 16:44:59.312268 6786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:40663 every 8 connection(s)
I20260928 16:44:59.313316 6655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:59.315944 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.300974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:60185 (local address 127.30.196.62:45841)
0928 16:44:59.301165 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.301172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.301997 (+ 825us) server_negotiation.cc:407] Connection header received
0928 16:44:59.303066 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.303071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.303136 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.303202 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.305021 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.305616 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.306795 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.307052 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.307175 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.311738 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.311743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.311745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.311777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.313491 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.314155 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.314160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.314162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.314275 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.314681 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.314684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.314685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.314889 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.314985 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.315565 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.315774 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:44:59.316881 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.301355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.1:60185)
0928 16:44:59.301820 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.301871 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.302816 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.303737 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.303745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.304209 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.304851 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.304873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.305749 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.305752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.306661 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.306671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.307222 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.307226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.307447 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.308272 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.308299 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.311466 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.313647 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.313653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.313673 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.314034 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.314412 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.314415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.314417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.314581 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.315031 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.315038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.315311 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.315492 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.315994 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260928 16:44:59.317767 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6655
I20260928 16:44:59.317935 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:59.318228 31504 external_mini_cluster.cc:1428] Setting key 47528abbef89898d0dfad8c4324226d2
I20260928 16:44:59.318928 6787 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45841
I20260928 16:44:59.319140 6787 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:59.319835 6787 heartbeater.cc:511] Master 127.30.196.62:45841 requested a full tablet report, sending...
I20260928 16:44:59.322715 6582 ts_manager.cc:194] Registered new tserver with Master: 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663)
I20260928 16:44:59.324342 6582 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:60185
WARNING: no policy specified for oryx/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:59.329591 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.320867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:48367 (local address 127.30.196.62:45841)
0928 16:44:59.321021 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.321026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.321280 (+ 254us) server_negotiation.cc:407] Connection header received
0928 16:44:59.321412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.321414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.321458 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.321525 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.322158 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.323019 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.324322 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.324620 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.324698 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.326249 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.326252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.326254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.326276 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.327866 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.328320 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.328324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.328326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.328414 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.328708 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.328713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.328715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.328912 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.329033 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.329302 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.329420 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:59.329638 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.320800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.1:48367)
0928 16:44:59.321182 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.321201 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.321318 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.321572 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.321578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.321851 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.322003 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.322012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.323230 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.323234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.324173 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.324186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.324862 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.324865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.324969 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.325604 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.325624 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.326095 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.327985 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.327990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.327992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.328210 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.328510 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.328514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.328516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.328615 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.329015 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.329018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.329129 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.329243 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.329409 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
I20260928 16:44:59.352308 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45841
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:59.467557 6796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:59.467900 6796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:59.467979 6796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:59.472198 6796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:59.472268 6796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:59.472308 6796 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:59.472373 6796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:44:59.477806 6796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:59.478940 6796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:59.480083 6796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:59.490342 6796 server_base.cc:1034] running on GCE node
W20260928 16:44:59.490288 6802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:59.490288 6804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:59.490290 6801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:59.490976 6796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:59.491778 6796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:59.492960 6796 hybrid_clock.cc:648] HybridClock initialized: now 1790613899492928 us; error 43 us; skew 500 ppm
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:59.495913 6796 init.cc:377] Logged in from keytab as oryx/127.30.196.2@KRBTEST.COM (short username oryx)
I20260928 16:44:59.497336 6796 webserver.cc:460] Webserver started at http://127.30.196.2:45097/ using document root <none> and password file <none>
I20260928 16:44:59.497951 6796 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:59.498015 6796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:59.498241 6796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:59.500464 6796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "d6769015f71d43edbcd5916a18c2bf00"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "3e0abfd6cc7b04acd7c2d33706c5f5f3"
server_key_iv: "fd75a05ee9f16f00d8e546de3b93ff2d"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.501029 6796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "d6769015f71d43edbcd5916a18c2bf00"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "3e0abfd6cc7b04acd7c2d33706c5f5f3"
server_key_iv: "fd75a05ee9f16f00d8e546de3b93ff2d"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.505126 6796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:59.507632 6811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.509196 6796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:44:59.509330 6796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "d6769015f71d43edbcd5916a18c2bf00"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "3e0abfd6cc7b04acd7c2d33706c5f5f3"
server_key_iv: "fd75a05ee9f16f00d8e546de3b93ff2d"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.509451 6796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:59.541806 6796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:59.545042 6796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:59.545213 6796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:59.546741 6796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:59.546794 6796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.546833 6796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:59.546896 6796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:59.558348 6796 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:45703
I20260928 16:44:59.558409 6927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:45703 every 8 connection(s)
I20260928 16:44:59.559370 6796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:59.559928 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6796
I20260928 16:44:59.560048 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:59.560386 31504 external_mini_cluster.cc:1428] Setting key 142095fce6512e86fde8f91d2cefdfd9
I20260928 16:44:59.564535 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.547925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:53551 (local address 127.30.196.62:45841)
0928 16:44:59.548100 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.548105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.549138 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 16:44:59.550602 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.550608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.550674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.550794 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.553210 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.553993 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.555451 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.555780 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.555901 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.559074 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.559078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.559080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.559106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.561962 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.562638 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.562640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.562642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.562721 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.563181 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.563184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.563185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.563375 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.563452 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.564222 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.564365 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:59.565428 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.548366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.2:53551)
0928 16:44:59.548934 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.548996 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.550307 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.550999 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.551008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.551561 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.552536 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.552561 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.554141 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.554144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.555322 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.555333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.555946 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.555950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.556141 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.556840 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.556863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.558867 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.562097 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.562104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.562120 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.562528 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.562821 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.562828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.562832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.563066 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.563507 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.563517 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.563910 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.564178 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.564577 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":112,"threads_started":1}
I20260928 16:44:59.566872 6928 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45841
I20260928 16:44:59.567121 6928 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:59.567764 6928 heartbeater.cc:511] Master 127.30.196.62:45841 requested a full tablet report, sending...
I20260928 16:44:59.569046 6583 ts_manager.cc:194] Registered new tserver with Master: d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703)
I20260928 16:44:59.569936 6583 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:53551
WARNING: no policy specified for oryx/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.3@KRBTEST.COM" created.
I20260928 16:44:59.579205 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.571742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.2:60507)
0928 16:44:59.572061 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.572073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.572171 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.572473 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.572476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.572653 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.572749 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.572754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.573560 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.573563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.574232 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.574243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.574768 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.574771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.574845 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.575448 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.575460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.575859 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.577604 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.577609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.577612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.577822 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.578112 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.578116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.578118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.578205 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.578631 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.578635 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.578742 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.578850 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.579007 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260928 16:44:59.579198 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.571805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:60507 (local address 127.30.196.62:45841)
0928 16:44:59.571946 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.571951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.572185 (+ 234us) server_negotiation.cc:407] Connection header received
0928 16:44:59.572308 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.572313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.572358 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.572418 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.572880 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.573409 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.574355 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.574638 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.574755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.576007 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.576010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.576015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.576047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.577486 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.577926 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.577943 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.577945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.578017 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.578296 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.578301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.578302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.578532 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.578642 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.578906 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.579031 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
I20260928 16:44:59.595109 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45841
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:59.715498 6937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:59.715849 6937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:59.715899 6937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:59.720094 6937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:59.720155 6937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:59.720197 6937 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:59.720232 6937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:44:59.726327 6937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.196.62:45841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:44:59.727384 6937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:59.728461 6937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:59.738829 6945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:59.738961 6937 server_base.cc:1034] running on GCE node
W20260928 16:44:59.738869 6942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:59.738869 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:59.739516 6937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:59.740259 6937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:59.741473 6937 hybrid_clock.cc:648] HybridClock initialized: now 1790613899741438 us; error 48 us; skew 500 ppm
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:59.744270 6937 init.cc:377] Logged in from keytab as oryx/127.30.196.3@KRBTEST.COM (short username oryx)
I20260928 16:44:59.745096 6937 webserver.cc:460] Webserver started at http://127.30.196.3:41257/ using document root <none> and password file <none>
I20260928 16:44:59.745636 6937 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:59.745697 6937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:59.745858 6937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:59.747977 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "d503035438ba4a33ae119110bb1e714f"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "c6187e693a22c22fa59b39d50b77a784"
server_key_iv: "db9895e4d87d441897fb9122eefb8a8e"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.748502 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "d503035438ba4a33ae119110bb1e714f"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "c6187e693a22c22fa59b39d50b77a784"
server_key_iv: "db9895e4d87d441897fb9122eefb8a8e"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.752393 6937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:59.754967 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.756500 6937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:44:59.756637 6937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "d503035438ba4a33ae119110bb1e714f"
format_stamp: "Formatted at 2026-09-28 16:44:59 on dist-test-slave-5rg1"
server_key: "c6187e693a22c22fa59b39d50b77a784"
server_key_iv: "db9895e4d87d441897fb9122eefb8a8e"
server_key_version: "encryptionkey@0"
I20260928 16:44:59.756732 6937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:59.783515 6937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:59.787191 6937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:59.787364 6937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:59.788887 6937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:59.788962 6937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:59.789002 6937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:59.789017 6937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:59.800388 6937 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:33477
I20260928 16:44:59.800470 7067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:33477 every 8 connection(s)
I20260928 16:44:59.801527 6937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:59.802732 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 6937
I20260928 16:44:59.803009 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:59.803371 31504 external_mini_cluster.cc:1428] Setting key ec3254431008e8058fb113ff215d8dae
I20260928 16:44:59.804220 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.790013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:38859 (local address 127.30.196.62:45841)
0928 16:44:59.790177 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.790183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.791033 (+ 850us) server_negotiation.cc:407] Connection header received
0928 16:44:59.792229 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.792234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.792300 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.792361 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.793984 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.794820 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.796761 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.797104 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.797193 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.800299 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.800303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.800312 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.800372 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.801799 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.802366 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.802370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.802372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.802489 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.802847 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.802851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.802853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.803067 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.803186 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.803956 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.804095 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:59.805193 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.790390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.3:38859)
0928 16:44:59.790862 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.790914 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.791966 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.792820 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.792830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.793170 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.793833 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.793848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.794970 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.794975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.796600 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.796617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.797286 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.797290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.797521 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.798267 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.798289 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.800111 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.801918 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.801926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.801940 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.802265 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.802597 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.802600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.802602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.802738 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.803197 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.803204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.803562 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.803845 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.804197 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
I20260928 16:44:59.806434 7068 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45841
I20260928 16:44:59.806653 7068 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:59.807186 7068 heartbeater.cc:511] Master 127.30.196.62:45841 requested a full tablet report, sending...
I20260928 16:44:59.808279 6583 ts_manager.cc:194] Registered new tserver with Master: d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477)
I20260928 16:44:59.809072 6583 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:38859
I20260928 16:44:59.816216 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.808939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:41801 (local address 127.30.196.62:45841)
0928 16:44:59.809133 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.809139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.809245 (+ 106us) server_negotiation.cc:407] Connection header received
0928 16:44:59.809382 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.809388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.809448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.809500 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.810021 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.810557 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.811394 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.811628 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.811779 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.812933 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.812937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.812940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.812967 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.814472 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.814955 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.814959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.814961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.815047 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.815327 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.815332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.815334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.815525 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.815636 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.815938 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.816057 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:59.816228 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.808809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.30.196.3:41801)
0928 16:44:59.809152 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.809166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.809266 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.809630 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.809632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.809809 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:59.809905 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.809910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.810694 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.810698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.811271 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.811280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.811786 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.811790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.811880 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.812361 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:59.812372 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:59.812784 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:59.814637 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.814640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.814643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.814847 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.815141 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:59.815145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:59.815147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:59.815236 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:59.815623 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:59.815626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:59.815742 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.815860 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.816027 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260928 16:44:59.817694 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:59.842898 6632 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:59.835869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.30.196.62:45841)
0928 16:44:59.836075 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.836079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.836096 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:59.836155 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.836158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.836200 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.836327 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.837134 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.837685 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.838587 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.838807 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.838926 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.842755 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45528: BlockingRecv error: recv got EOF from 127.0.0.1:45528 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 16:44:59 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613899, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:44:59.853645 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.844662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.30.196.62:45841)
0928 16:44:59.844793 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.844797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.844891 (+ 94us) server_negotiation.cc:407] Connection header received
0928 16:44:59.844992 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.844995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.845042 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.845125 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:59.845779 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.846300 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.847181 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.847460 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.847557 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.850162 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:59.850166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:59.850168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:59.850196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:59.851857 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.852358 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.852362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.852363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.852448 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:59.852735 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:59.852738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:59.852739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:59.853027 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:59.853152 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.853394 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.853512 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:44:59.857458 6582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:59.860950 6582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:59.876482 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.870131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33477 (local address 127.0.0.1:34522)
0928 16:44:59.870545 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.870601 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.870735 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.871618 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.871623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.871645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:59.871838 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.871847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873394 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.873397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.874721 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.874732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.875863 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.875866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.875980 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.876024 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.876186 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.876317 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":131,"threads_started":1}
I20260928 16:44:59.876609 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.870995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.30.196.2:45703)
0928 16:44:59.871320 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.871333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.871694 (+ 361us) server_negotiation.cc:407] Connection header received
0928 16:44:59.871780 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.871785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.871962 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.872119 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:59.872537 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873571 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.875196 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.876016 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.876121 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.876267 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.876357 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.876452 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":150,"threads_started":1}
I20260928 16:44:59.876844 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.870374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45703 (local address 127.0.0.1:44006)
0928 16:44:59.871415 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.871428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.871557 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.872272 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.872276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.872293 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:59.872395 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.872401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873698 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.873702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.875052 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.875065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.876124 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.876128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.876216 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.876240 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.876322 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.876398 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":74,"thread_start_us":145,"threads_started":1}
I20260928 16:44:59.877012 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.870298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.30.196.3:33477)
0928 16:44:59.871069 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.871082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.871110 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:44:59.871209 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.871215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.871432 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.872009 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:59.872182 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873265 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.874859 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.875727 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.875846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.876031 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.876564 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.876667 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":116,"threads_started":1}
I20260928 16:44:59.877305 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.871052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.0.0.1:59670)
0928 16:44:59.871936 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:59.871950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:59.872023 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:59.872647 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:59.872650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:59.872667 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:59.872767 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.872776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873847 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.873850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.875060 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:59.875065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.876900 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.876906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:59.877018 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.877039 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:44:59.877110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:44:59.877183 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
I20260928 16:44:59.877825 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:59.871578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.30.196.1:40663)
0928 16:44:59.872213 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:59.872221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:59.872243 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:59.872316 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:59.872320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:59.872505 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:59.872670 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:59.872863 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.873729 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.875765 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:59.876583 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:59.876718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:59.876839 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:59.877526 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 16:44:59.877664 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":122,"threads_started":1}
I20260928 16:44:59.879926 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 (DEFAULT_TABLE table=test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:59.880144 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 (DEFAULT_TABLE table=test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:59.880997 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d87dd5ff9d24f81b1ebb733466db054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:59.881021 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d87dd5ff9d24f81b1ebb733466db054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:59.882306 6720 tablet_service.cc:1467] Processing CreateTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 (DEFAULT_TABLE table=test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:59.883317 6720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d87dd5ff9d24f81b1ebb733466db054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:59.888981 7097 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Bootstrap starting.
I20260928 16:44:59.889568 7096 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Bootstrap starting.
I20260928 16:44:59.890621 7098 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Bootstrap starting.
I20260928 16:44:59.891501 7097 tablet_bootstrap.cc:654] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:59.891616 7096 tablet_bootstrap.cc:654] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:59.892378 7096 log.cc:824] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:59.892426 7097 log.cc:824] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:59.892817 7098 tablet_bootstrap.cc:654] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:59.893597 7098 log.cc:824] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:59.894577 7096 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: No bootstrap required, opened a new log
I20260928 16:44:59.894575 7097 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: No bootstrap required, opened a new log
I20260928 16:44:59.894785 7096 ts_tablet_manager.cc:1397] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:59.894785 7097 ts_tablet_manager.cc:1397] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:44:59.895314 7098 tablet_bootstrap.cc:492] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: No bootstrap required, opened a new log
I20260928 16:44:59.895531 7098 ts_tablet_manager.cc:1397] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:44:59.898939 7098 raft_consensus.cc:348] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.898963 7097 raft_consensus.cc:348] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.898963 7096 raft_consensus.cc:348] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.899230 7098 raft_consensus.cc:374] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:59.899230 7096 raft_consensus.cc:374] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:59.899230 7097 raft_consensus.cc:374] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:59.899296 7098 raft_consensus.cc:729] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Initialized, Role: FOLLOWER
I20260928 16:44:59.899299 7096 raft_consensus.cc:729] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Initialized, Role: FOLLOWER
I20260928 16:44:59.899572 7097 raft_consensus.cc:729] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Initialized, Role: FOLLOWER
I20260928 16:44:59.900101 7098 consensus_queue.cc:260] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.900101 7097 consensus_queue.cc:260] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.900151 7096 consensus_queue.cc:260] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:44:59.901172 6928 heartbeater.cc:503] Master 127.30.196.62:45841 was elected leader, sending a full tablet report...
I20260928 16:44:59.901228 7098 ts_tablet_manager.cc:1428] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:44:59.901228 7097 ts_tablet_manager.cc:1428] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:44:59.901569 6787 heartbeater.cc:503] Master 127.30.196.62:45841 was elected leader, sending a full tablet report...
I20260928 16:44:59.901602 7068 heartbeater.cc:503] Master 127.30.196.62:45841 was elected leader, sending a full tablet report...
I20260928 16:44:59.902544 7096 ts_tablet_manager.cc:1428] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 16:45:00.052582 7069 tablet.cc:2369] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:45:00.060623 6929 tablet.cc:2369] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:45:00.065037 6788 tablet.cc:2369] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:45:00.136725 7103 raft_consensus.cc:482] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:00.136934 7103 raft_consensus.cc:504] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:45:00.138101 7103 leader_election.cc:288] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703), 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663)
I20260928 16:45:00.141908 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.138422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.30.196.3:48493)
0928 16:45:00.138546 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.138565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.138681 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.139006 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.139009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.139028 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:00.139152 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.139158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.139942 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.139947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.140740 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.140750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.141461 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.141464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.141574 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.141593 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.141689 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.141769 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 16:45:00.141908 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.138643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:48493 (local address 127.30.196.1:40663)
0928 16:45:00.138785 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.138789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.138810 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:45:00.138859 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.138861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.138898 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.138992 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:00.139276 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.139822 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.140874 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.141331 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.141467 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.141522 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.141706 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.141782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:45:00.142688 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d87dd5ff9d24f81b1ebb733466db054" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" is_pre_election: true
I20260928 16:45:00.143059 6741 raft_consensus.cc:2454] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:00.143235 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.138488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45703 (local address 127.30.196.3:60329)
0928 16:45:00.138925 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.138938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.139083 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.139381 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.139384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.139401 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:00.140045 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.140056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.140872 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.140879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.141706 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.141715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.142707 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.142711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.142828 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.142879 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.143003 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.143079 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":164,"threads_started":1}
I20260928 16:45:00.143219 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.138564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:60329 (local address 127.30.196.2:45703)
0928 16:45:00.138724 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.138729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.139042 (+ 313us) server_negotiation.cc:407] Connection header received
0928 16:45:00.139217 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.139220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.139272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.139359 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:00.140183 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.140747 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.141841 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.142546 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.142691 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.142769 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.143022 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.143098 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:45:00.143622 6953 leader_election.cc:302] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895587bd60f34bd4a311a8e6ba6f3bc8, d503035438ba4a33ae119110bb1e714f; no voters:
I20260928 16:45:00.144037 7103 raft_consensus.cc:2781] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:00.144011 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d87dd5ff9d24f81b1ebb733466db054" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00" is_pre_election: true
I20260928 16:45:00.144141 7103 raft_consensus.cc:482] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:00.144191 7103 raft_consensus.cc:3037] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.144307 6882 raft_consensus.cc:2454] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:00.145421 7103 raft_consensus.cc:504] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:45:00.145843 7103 leader_election.cc:288] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Requested vote from peers d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703), 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663)
I20260928 16:45:00.146210 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d87dd5ff9d24f81b1ebb733466db054" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00"
I20260928 16:45:00.146364 6882 raft_consensus.cc:3037] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.146449 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d87dd5ff9d24f81b1ebb733466db054" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
I20260928 16:45:00.146617 6741 raft_consensus.cc:3037] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.147547 6882 raft_consensus.cc:2454] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:00.147882 6741 raft_consensus.cc:2454] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:00.147917 6955 leader_election.cc:302] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d503035438ba4a33ae119110bb1e714f, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:00.148087 7103 raft_consensus.cc:2781] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:00.148303 7103 raft_consensus.cc:686] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Becoming Leader. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Running, Role: LEADER
I20260928 16:45:00.148686 7103 consensus_queue.cc:237] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } }
I20260928 16:45:00.152487 6582 catalog_manager.cc:5495] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f reported cstate change: term changed from 0 to 1, leader changed from <none> to d503035438ba4a33ae119110bb1e714f (127.30.196.3). New cstate: current_term: 1 leader_uuid: "d503035438ba4a33ae119110bb1e714f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } health_report { overall_health: UNKNOWN } } }
I20260928 16:45:00.170873 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.167045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.30.196.3:33477)
0928 16:45:00.167180 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.167185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.167201 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:45:00.167390 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.167393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.167450 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.167566 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:00.167851 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.168518 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.169389 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.169621 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.169735 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.170014 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.170463 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.170595 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.170665 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.170725 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 16:45:00.176954 6882 raft_consensus.cc:1261] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.177021 6741 raft_consensus.cc:1261] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.177798 7106 consensus_queue.cc:1035] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.178021 7103 consensus_queue.cc:1035] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.188041 7110 mvcc.cc:205] Tried to move back new op lower bound from 7334354535116894208 to 7334354535012474880. Current Snapshot: MvccSnapshot[applied={T|T < 7334354535116894208}]
I20260928 16:45:00.188969 7111 mvcc.cc:205] Tried to move back new op lower bound from 7334354535116894208 to 7334354535012474880. Current Snapshot: MvccSnapshot[applied={T|T < 7334354535116894208}]
I20260928 16:45:00.197000 6581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:00.197211 6581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:00.199883 6581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Sending DeleteTablet for 3 replicas of tablet 9d87dd5ff9d24f81b1ebb733466db054
I20260928 16:45:00.200752 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:34522
I20260928 16:45:00.200747 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44006
I20260928 16:45:00.201303 6720 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d87dd5ff9d24f81b1ebb733466db054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 16:45:00.201735 7119 tablet_replica.cc:330] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: stopping tablet replica
I20260928 16:45:00.201802 7118 tablet_replica.cc:330] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: stopping tablet replica
I20260928 16:45:00.201856 6581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:45:00.202153 7118 raft_consensus.cc:2229] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 16:45:00.202319 6581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:00.202441 7119 raft_consensus.cc:2229] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:00.202498 7118 raft_consensus.cc:2258] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.202979 7119 raft_consensus.cc:2258] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.204147 7118 ts_tablet_manager.cc:1905] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.204948 7119 ts_tablet_manager.cc:1905] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.207235 7118 ts_tablet_manager.cc:1918] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:00.207388 7118 log.cc:1196] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/9d87dd5ff9d24f81b1ebb733466db054
I20260928 16:45:00.207769 7118 ts_tablet_manager.cc:1939] T 9d87dd5ff9d24f81b1ebb733466db054 P d6769015f71d43edbcd5916a18c2bf00: Deleting consensus metadata
I20260928 16:45:00.208254 7119 ts_tablet_manager.cc:1918] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:00.208362 7119 log.cc:1196] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/9d87dd5ff9d24f81b1ebb733466db054
I20260928 16:45:00.208456 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (DEFAULT_TABLE table=test-table [id=852beebf37eb45398eca0a0f0bae35b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.208729 7119 ts_tablet_manager.cc:1939] T 9d87dd5ff9d24f81b1ebb733466db054 P d503035438ba4a33ae119110bb1e714f: Deleting consensus metadata
I20260928 16:45:00.208737 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.209175 6568 catalog_manager.cc:4841] TS d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703): tablet 9d87dd5ff9d24f81b1ebb733466db054 (table test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]) successfully deleted
I20260928 16:45:00.209628 6720 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (DEFAULT_TABLE table=test-table [id=852beebf37eb45398eca0a0f0bae35b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.209717 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (DEFAULT_TABLE table=test-table [id=852beebf37eb45398eca0a0f0bae35b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.209983 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.209998 6720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.211133 7096 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Bootstrap starting.
I20260928 16:45:00.212271 7096 tablet_bootstrap.cc:654] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.212332 7097 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Bootstrap starting.
I20260928 16:45:00.212643 7098 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Bootstrap starting.
I20260928 16:45:00.213330 7097 tablet_bootstrap.cc:654] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.213814 7098 tablet_bootstrap.cc:654] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.214159 7120 tablet_replica.cc:330] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: stopping tablet replica
I20260928 16:45:00.214525 7120 raft_consensus.cc:2229] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:00.214870 7120 raft_consensus.cc:2258] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.215813 6566 catalog_manager.cc:4841] TS d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477): tablet 9d87dd5ff9d24f81b1ebb733466db054 (table test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]) successfully deleted
I20260928 16:45:00.215957 7097 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: No bootstrap required, opened a new log
I20260928 16:45:00.216042 7097 ts_tablet_manager.cc:1397] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:45:00.216156 7098 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: No bootstrap required, opened a new log
I20260928 16:45:00.216231 7098 ts_tablet_manager.cc:1397] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:45:00.216606 7097 raft_consensus.cc:348] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.216667 7098 raft_consensus.cc:348] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.216758 7097 raft_consensus.cc:374] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.216771 7098 raft_consensus.cc:374] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.216800 7098 raft_consensus.cc:729] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.216800 7097 raft_consensus.cc:729] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.216833 7096 tablet_bootstrap.cc:492] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: No bootstrap required, opened a new log
I20260928 16:45:00.216931 7096 ts_tablet_manager.cc:1397] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260928 16:45:00.216919 7098 consensus_queue.cc:260] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.216956 7097 consensus_queue.cc:260] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.217176 7098 ts_tablet_manager.cc:1428] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:45:00.217227 7097 ts_tablet_manager.cc:1428] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:45:00.217418 7096 raft_consensus.cc:348] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.217566 7096 raft_consensus.cc:374] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.217602 7096 raft_consensus.cc:729] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.217772 7096 consensus_queue.cc:260] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.218214 7096 ts_tablet_manager.cc:1428] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 16:45:00.219336 7120 ts_tablet_manager.cc:1905] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.222757 7120 ts_tablet_manager.cc:1918] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:00.222843 7120 log.cc:1196] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/9d87dd5ff9d24f81b1ebb733466db054
I20260928 16:45:00.223134 7120 ts_tablet_manager.cc:1939] T 9d87dd5ff9d24f81b1ebb733466db054 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting consensus metadata
I20260928 16:45:00.224368 6566 catalog_manager.cc:4841] TS 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663): tablet 9d87dd5ff9d24f81b1ebb733466db054 (table test-table [id=2e3f5ce6e203464cbe1edd2731a513f8]) successfully deleted
I20260928 16:45:00.336730 7103 raft_consensus.cc:482] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:00.336891 7103 raft_consensus.cc:504] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.337399 7103 leader_election.cc:288] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703)
I20260928 16:45:00.337944 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2d2930b9b942cdbd7e9fdc834d00a2" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00" is_pre_election: true
I20260928 16:45:00.337931 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2d2930b9b942cdbd7e9fdc834d00a2" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" is_pre_election: true
I20260928 16:45:00.338100 6882 raft_consensus.cc:2454] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:00.338100 6741 raft_consensus.cc:2454] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:00.338421 6953 leader_election.cc:302] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895587bd60f34bd4a311a8e6ba6f3bc8, d503035438ba4a33ae119110bb1e714f; no voters:
I20260928 16:45:00.338595 7103 raft_consensus.cc:2781] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:00.338675 7103 raft_consensus.cc:482] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:00.338728 7103 raft_consensus.cc:3037] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.340247 7103 raft_consensus.cc:504] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.340690 7103 leader_election.cc:288] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Requested vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703)
I20260928 16:45:00.341063 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2d2930b9b942cdbd7e9fdc834d00a2" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
I20260928 16:45:00.341205 6741 raft_consensus.cc:3037] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.341254 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2d2930b9b942cdbd7e9fdc834d00a2" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00"
I20260928 16:45:00.341418 6882 raft_consensus.cc:3037] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.342427 6741 raft_consensus.cc:2454] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:00.342453 6882 raft_consensus.cc:2454] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:00.342799 6955 leader_election.cc:302] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d503035438ba4a33ae119110bb1e714f, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:00.343039 7103 raft_consensus.cc:2781] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:00.343186 7103 raft_consensus.cc:686] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Becoming Leader. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Running, Role: LEADER
I20260928 16:45:00.343348 7103 consensus_queue.cc:237] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.345224 6581 catalog_manager.cc:5495] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f reported cstate change: term changed from 0 to 1, leader changed from <none> to d503035438ba4a33ae119110bb1e714f (127.30.196.3). New cstate: current_term: 1 leader_uuid: "d503035438ba4a33ae119110bb1e714f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } health_report { overall_health: HEALTHY } } }
Sep 28 16:45:00 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613898, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.62@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:45:00.402143 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.393532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.30.196.62:45841)
0928 16:45:00.393765 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.393768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.393783 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:00.393884 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.393886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.393928 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.394009 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:00.394722 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.395246 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.396147 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.396362 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.396442 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.398772 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:00.398776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:00.398778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:00.398798 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:00.400387 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:00.400869 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:00.400875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:00.400876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:00.400958 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:00.401235 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:00.401238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:00.401240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:00.401415 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:00.401522 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.401819 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.401988 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20260928 16:45:00.402143 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.393400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45841 (local address 127.0.0.1:45556)
0928 16:45:00.393661 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.393679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.393795 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.394097 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.394100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.394513 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:00.394612 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.394618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.395348 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.395350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.396022 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.396032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.396477 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.396480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.396544 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.397103 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:00.397115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:00.398603 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:00.400511 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:00.400521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:00.400528 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:00.400768 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:00.401048 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:00.401054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:00.401056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:00.401140 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:00.401521 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:00.401529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:00.401648 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.401764 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.401929 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260928 16:45:00.406397 6581 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:45556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:45:00.407187 6581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:00.412952 6720 tablet_service.cc:1467] Processing CreateTablet for tablet 3c40302713cd4dc68ffc44236647b0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a117034a6c6f42a4b7f88deda8796495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:45:00.412971 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 3c40302713cd4dc68ffc44236647b0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a117034a6c6f42a4b7f88deda8796495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:45:00.413436 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c40302713cd4dc68ffc44236647b0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.413445 6720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c40302713cd4dc68ffc44236647b0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.416199 7096 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: Bootstrap starting.
I20260928 16:45:00.416147 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 3c40302713cd4dc68ffc44236647b0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a117034a6c6f42a4b7f88deda8796495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:45:00.416584 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c40302713cd4dc68ffc44236647b0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.417102 7096 tablet_bootstrap.cc:654] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.417709 7098 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Bootstrap starting.
I20260928 16:45:00.418314 7096 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: No bootstrap required, opened a new log
I20260928 16:45:00.418397 7096 ts_tablet_manager.cc:1397] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 16:45:00.418591 7098 tablet_bootstrap.cc:654] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.418848 7096 raft_consensus.cc:348] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.418967 7096 raft_consensus.cc:374] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.419003 7096 raft_consensus.cc:729] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.419133 7096 consensus_queue.cc:260] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.419399 7096 ts_tablet_manager.cc:1428] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 16:45:00.419639 7097 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: Bootstrap starting.
I20260928 16:45:00.419613 7104 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.419782 7104 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.419920 7098 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: No bootstrap required, opened a new log
I20260928 16:45:00.420019 7098 ts_tablet_manager.cc:1397] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:00.420429 7097 tablet_bootstrap.cc:654] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.420405 7098 raft_consensus.cc:348] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.420536 7098 raft_consensus.cc:374] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.420578 7098 raft_consensus.cc:729] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.420710 7098 consensus_queue.cc:260] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.420949 7102 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.421084 7102 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.421558 7097 tablet_bootstrap.cc:492] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: No bootstrap required, opened a new log
I20260928 16:45:00.421658 7097 ts_tablet_manager.cc:1397] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:00.422076 7098 ts_tablet_manager.cc:1428] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:45:00.422084 7097 raft_consensus.cc:348] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.422199 7097 raft_consensus.cc:374] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.422243 7097 raft_consensus.cc:729] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.422389 7097 consensus_queue.cc:260] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.422652 7103 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.422794 7103 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.422658 7097 ts_tablet_manager.cc:1428] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:45:00.526499 7102 raft_consensus.cc:482] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:00.526647 7102 raft_consensus.cc:504] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.527791 7102 leader_election.cc:288] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703), d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477)
I20260928 16:45:00.531831 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.528311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:39753 (local address 127.30.196.2:45703)
0928 16:45:00.528448 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.528461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.528532 (+ 71us) server_negotiation.cc:407] Connection header received
0928 16:45:00.528706 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.528709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.528759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.528842 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:00.529183 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.529709 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.530811 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.531201 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.531333 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.531392 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.531596 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.531696 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:45:00.531858 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.528222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:54137 (local address 127.30.196.3:33477)
0928 16:45:00.528461 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.528464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.528712 (+ 248us) server_negotiation.cc:407] Connection header received
0928 16:45:00.528866 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.528871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.528928 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.529011 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:00.529284 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.529775 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.530815 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.531220 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.531347 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.531404 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.531636 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.531730 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260928 16:45:00.531872 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.528036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:45703 (local address 127.30.196.1:39753)
0928 16:45:00.528458 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.528474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.528588 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.528855 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.528858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.528875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:00.529070 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.529077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.529821 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.529826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.530687 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.530696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.531330 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.531333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.531463 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.531491 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.531579 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.531653 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260928 16:45:00.531879 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.528043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33477 (local address 127.30.196.1:54137)
0928 16:45:00.528610 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.528630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.528757 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.529032 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.529035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.529064 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:00.529179 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.529184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.529906 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.529909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.530695 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.530703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.531347 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.531350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.531510 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.531527 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.531625 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.531720 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":101,"thread_start_us":140,"threads_started":1}
I20260928 16:45:00.532362 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c40302713cd4dc68ffc44236647b0b2" candidate_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00" is_pre_election: true
I20260928 16:45:00.532519 6882 raft_consensus.cc:2454] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895587bd60f34bd4a311a8e6ba6f3bc8 in term 0.
I20260928 16:45:00.532920 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c40302713cd4dc68ffc44236647b0b2" candidate_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d503035438ba4a33ae119110bb1e714f" is_pre_election: true
I20260928 16:45:00.532943 6673 leader_election.cc:302] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895587bd60f34bd4a311a8e6ba6f3bc8, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:00.533164 7022 raft_consensus.cc:2454] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895587bd60f34bd4a311a8e6ba6f3bc8 in term 0.
I20260928 16:45:00.533223 7102 raft_consensus.cc:2781] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:00.533274 7102 raft_consensus.cc:482] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:00.533313 7102 raft_consensus.cc:3037] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.534340 7102 raft_consensus.cc:504] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.534654 7102 leader_election.cc:288] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [CANDIDATE]: Term 1 election: Requested vote from peers d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703), d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477)
I20260928 16:45:00.535043 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c40302713cd4dc68ffc44236647b0b2" candidate_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00"
I20260928 16:45:00.535156 6882 raft_consensus.cc:3037] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.536222 6882 raft_consensus.cc:2454] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895587bd60f34bd4a311a8e6ba6f3bc8 in term 1.
I20260928 16:45:00.536523 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c40302713cd4dc68ffc44236647b0b2" candidate_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d503035438ba4a33ae119110bb1e714f"
I20260928 16:45:00.536549 6673 leader_election.cc:302] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895587bd60f34bd4a311a8e6ba6f3bc8, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:00.536652 7022 raft_consensus.cc:3037] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:00.536736 7102 raft_consensus.cc:2781] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:00.536968 7102 raft_consensus.cc:686] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Running, Role: LEADER
I20260928 16:45:00.537279 7102 consensus_queue.cc:237] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.537937 7022 raft_consensus.cc:2454] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895587bd60f34bd4a311a8e6ba6f3bc8 in term 1.
I20260928 16:45:00.539078 7137 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: TxnStatusTablet state changed. Reason: New leader 895587bd60f34bd4a311a8e6ba6f3bc8. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.539265 7137 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:45:00.539599 7139 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:45:00.539798 7139 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:45:00.540295 6581 catalog_manager.cc:5495] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1). New cstate: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } health_report { overall_health: UNKNOWN } } }
I20260928 16:45:00.554641 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.550910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.30.196.1:40663)
0928 16:45:00.551083 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.551087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.551102 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:00.551277 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.551280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.551350 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.551452 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:00.551730 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.552244 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.553114 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.553364 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.553485 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.553789 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.554206 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.554336 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.554413 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.554468 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:45:00.554641 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.550744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.0.0.1:59672)
0928 16:45:00.550979 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.550998 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.551114 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.551468 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.551471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.551479 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:45:00.551597 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.551603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.552360 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.552364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.552982 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.552992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.553485 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.553489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.553594 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.553639 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.554310 (+ 671us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.554326 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.554398 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.554492 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 16:45:00.558069 6882 raft_consensus.cc:1261] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Refusing update from remote peer 895587bd60f34bd4a311a8e6ba6f3bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.558465 7022 raft_consensus.cc:1261] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Refusing update from remote peer 895587bd60f34bd4a311a8e6ba6f3bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.558629 7137 consensus_queue.cc:1035] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.559230 7102 consensus_queue.cc:1035] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.560153 7104 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: TxnStatusTablet state changed. Reason: New leader 895587bd60f34bd4a311a8e6ba6f3bc8. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.560262 7104 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.562441 7103 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: TxnStatusTablet state changed. Reason: New leader 895587bd60f34bd4a311a8e6ba6f3bc8. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.562561 7103 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.565920 7109 mvcc.cc:205] Tried to move back new op lower bound from 7334354536680488960 to 7334354536603844608. Current Snapshot: MvccSnapshot[applied={T|T < 7334354536680488960}]
I20260928 16:45:00.566040 7103 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.566164 7103 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d503035438ba4a33ae119110bb1e714f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.568022 7102 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.568161 7102 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:45:00.570086 7102 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.570205 7102 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P 895587bd60f34bd4a311a8e6ba6f3bc8: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:45:00.571020 7104 tablet_replica.cc:439] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } } }
I20260928 16:45:00.571132 7104 tablet_replica.cc:442] T 3c40302713cd4dc68ffc44236647b0b2 P d6769015f71d43edbcd5916a18c2bf00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:45:00.586555 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.579849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:36835 (local address 127.30.196.1:40663)
0928 16:45:00.579981 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.579986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.583245 (+ 3259us) server_negotiation.cc:407] Connection header received
0928 16:45:00.583422 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.583425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.583489 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.583580 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:00.583862 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.584452 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.585417 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.585669 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.585789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.586092 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.586175 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.586257 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.586351 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.586406 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:45:00.586573 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.579758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.30.196.3:36835)
0928 16:45:00.583117 (+ 3359us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.583131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.583293 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.583593 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.583596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.583604 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:45:00.583745 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.583752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.584568 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.584573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.585294 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.585305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.585797 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.585800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.585889 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.585948 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.586264 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.586274 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.586340 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.586440 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3252,"thread_start_us":110,"threads_started":1}
I20260928 16:45:00.594827 6741 raft_consensus.cc:1261] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.594827 6882 raft_consensus.cc:1261] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:00.595378 7106 consensus_queue.cc:1035] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.595577 7103 consensus_queue.cc:1035] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:00.620729 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.616546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:34625 (local address 127.30.196.3:33477)
0928 16:45:00.616692 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.616697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.617000 (+ 303us) server_negotiation.cc:407] Connection header received
0928 16:45:00.617143 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.617146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.617204 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.617290 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:00.617616 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.618160 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.619409 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.619745 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.619872 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.620227 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.620314 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.620398 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.620516 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.620576 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:00.620745 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.616394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33477 (local address 127.30.196.1:34625)
0928 16:45:00.616892 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:00.616906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:00.617019 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:00.617313 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:00.617316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:00.617324 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:45:00.617465 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.617471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.618269 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.618273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.619294 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:00.619305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.619873 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.619877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:00.620001 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.620086 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:45:00.620412 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:45:00.620421 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:45:00.620479 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 16:45:00.620571 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":129,"threads_started":1}
I20260928 16:45:00.648989 6581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:00.649191 6581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:00.651422 6581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Sending DeleteTablet for 3 replicas of tablet 5f2d2930b9b942cdbd7e9fdc834d00a2
I20260928 16:45:00.652251 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44006
I20260928 16:45:00.652298 6720 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 16:45:00.652448 7118 tablet_replica.cc:330] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: stopping tablet replica
I20260928 16:45:00.652488 7120 tablet_replica.cc:330] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: stopping tablet replica
I20260928 16:45:00.652577 7118 raft_consensus.cc:2229] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:00.652594 7120 raft_consensus.cc:2229] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:00.652698 7118 raft_consensus.cc:2258] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.652709 7120 raft_consensus.cc:2258] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.653306 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:00 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:34522
I20260928 16:45:00.653517 7119 tablet_replica.cc:330] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: stopping tablet replica
I20260928 16:45:00.653633 7119 raft_consensus.cc:2229] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:00.653657 7118 ts_tablet_manager.cc:1905] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.653843 7119 raft_consensus.cc:2258] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:00.653976 7120 ts_tablet_manager.cc:1905] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.654618 7119 ts_tablet_manager.cc:1905] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:00.656827 7118 ts_tablet_manager.cc:1918] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:45:00.656845 7120 ts_tablet_manager.cc:1918] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:45:00.656898 7118 log.cc:1196] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/5f2d2930b9b942cdbd7e9fdc834d00a2
I20260928 16:45:00.656920 7120 log.cc:1196] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/5f2d2930b9b942cdbd7e9fdc834d00a2
I20260928 16:45:00.657177 7118 ts_tablet_manager.cc:1939] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d6769015f71d43edbcd5916a18c2bf00: Deleting consensus metadata
I20260928 16:45:00.657198 7120 ts_tablet_manager.cc:1939] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting consensus metadata
I20260928 16:45:00.657524 7119 ts_tablet_manager.cc:1918] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:45:00.657593 7119 log.cc:1196] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/5f2d2930b9b942cdbd7e9fdc834d00a2
I20260928 16:45:00.657837 7119 ts_tablet_manager.cc:1939] T 5f2d2930b9b942cdbd7e9fdc834d00a2 P d503035438ba4a33ae119110bb1e714f: Deleting consensus metadata
I20260928 16:45:00.658082 6568 catalog_manager.cc:4841] TS d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703): tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (table test-table [id=852beebf37eb45398eca0a0f0bae35b0]) successfully deleted
I20260928 16:45:00.658075 6566 catalog_manager.cc:4841] TS 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663): tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (table test-table [id=852beebf37eb45398eca0a0f0bae35b0]) successfully deleted
I20260928 16:45:00.660513 6566 catalog_manager.cc:4841] TS d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477): tablet 5f2d2930b9b942cdbd7e9fdc834d00a2 (table test-table [id=852beebf37eb45398eca0a0f0bae35b0]) successfully deleted
Sep 28 16:45:00 dist-test-slave-5rg1 krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:45:00 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:00 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:45:00.685076 7131 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:00.678223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.30.196.62:45841)
0928 16:45:00.678370 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.678376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.678578 (+ 202us) server_negotiation.cc:407] Connection header received
0928 16:45:00.678747 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.678751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.678805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.678894 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:00.679628 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.680231 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.681095 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.681340 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.681454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.684845 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45562: BlockingRecv error: recv got EOF from 127.0.0.1:45562 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 16:45:00 dist-test-slave-5rg1 krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM
I20260928 16:45:00.696228 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:00.686732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.30.196.62:45841)
0928 16:45:00.686853 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:00.686857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:00.687204 (+ 347us) server_negotiation.cc:407] Connection header received
0928 16:45:00.687365 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:00.687368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:00.687424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:00.687509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:00.688250 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.688856 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.689706 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:00.689949 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:00.690081 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:00.692921 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:00.692925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:00.692927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:00.692955 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:00.694442 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:00.694906 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:00.694911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:00.694913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:00.695021 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:00.695302 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:00.695306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:00.695308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:00.695503 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:00.695647 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:00.695916 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:45:00.696053 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:45:00.699150 6581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:45:00.699646 6581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:00.704622 6720 tablet_service.cc:1467] Processing CreateTablet for tablet 1289cee6bab641ada0eeb54c85047604 (DEFAULT_TABLE table=test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.704841 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 1289cee6bab641ada0eeb54c85047604 (DEFAULT_TABLE table=test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.704841 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 1289cee6bab641ada0eeb54c85047604 (DEFAULT_TABLE table=test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:00.704957 6720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1289cee6bab641ada0eeb54c85047604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.705122 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1289cee6bab641ada0eeb54c85047604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.705122 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1289cee6bab641ada0eeb54c85047604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:00.707535 7096 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Bootstrap starting.
I20260928 16:45:00.707547 7098 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Bootstrap starting.
I20260928 16:45:00.708388 7096 tablet_bootstrap.cc:654] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.708570 7098 tablet_bootstrap.cc:654] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.709143 7097 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Bootstrap starting.
I20260928 16:45:00.710227 7098 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: No bootstrap required, opened a new log
I20260928 16:45:00.710314 7098 ts_tablet_manager.cc:1397] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:45:00.710592 7097 tablet_bootstrap.cc:654] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:00.710775 7098 raft_consensus.cc:348] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.710927 7098 raft_consensus.cc:374] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.710976 7098 raft_consensus.cc:729] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.711140 7098 consensus_queue.cc:260] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.711458 7096 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: No bootstrap required, opened a new log
I20260928 16:45:00.711453 7098 ts_tablet_manager.cc:1428] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:45:00.711566 7096 ts_tablet_manager.cc:1397] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:45:00.712019 7096 raft_consensus.cc:348] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.712149 7096 raft_consensus.cc:374] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.712189 7096 raft_consensus.cc:729] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.712325 7096 consensus_queue.cc:260] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.712600 7096 ts_tablet_manager.cc:1428] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:45:00.713531 7097 tablet_bootstrap.cc:492] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: No bootstrap required, opened a new log
I20260928 16:45:00.713650 7097 ts_tablet_manager.cc:1397] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:45:00.714089 7097 raft_consensus.cc:348] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.714196 7097 raft_consensus.cc:374] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:00.714236 7097 raft_consensus.cc:729] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Initialized, Role: FOLLOWER
I20260928 16:45:00.714372 7097 consensus_queue.cc:260] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:00.714659 7097 ts_tablet_manager.cc:1428] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:45:01.005471 7104 raft_consensus.cc:482] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:01.005745 7104 raft_consensus.cc:504] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.006896 7104 leader_election.cc:288] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477)
I20260928 16:45:01.011821 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.007288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:36681 (local address 127.30.196.1:40663)
0928 16:45:01.007447 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:01.007451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:01.007568 (+ 117us) server_negotiation.cc:407] Connection header received
0928 16:45:01.007805 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:01.007809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:01.007870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:01.007977 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:01.008384 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.009221 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.010638 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.011169 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.011263 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.011307 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:01.011572 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 16:45:01.011650 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:45:01.011852 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.007171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.30.196.2:36681)
0928 16:45:01.007461 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:01.007478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:01.007598 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:01.008004 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:01.008009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:01.008042 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:01.008253 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:01.008261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.009345 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.009349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:01.010499 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:01.010510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.011309 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.011313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:01.011410 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.011439 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:45:01.011528 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:45:01.011608 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260928 16:45:01.012065 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.007729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33319 (local address 127.30.196.3:33477)
0928 16:45:01.007865 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:01.007869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:01.008584 (+ 715us) server_negotiation.cc:407] Connection header received
0928 16:45:01.008704 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:01.008706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:01.008753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:01.008819 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:01.009132 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.009837 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.010958 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.011460 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.011573 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.011627 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:01.011843 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 16:45:01.011932 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:45:01.012385 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.007585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:33477 (local address 127.30.196.2:33319)
0928 16:45:01.008484 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:01.008499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:01.008603 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:01.008864 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:01.008868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:01.008885 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:01.009013 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:01.009021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.009943 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.009945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:01.010827 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:01.010838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.011583 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.011587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:01.011707 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.011732 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:45:01.011810 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:45:01.012258 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":640,"threads_started":1}
I20260928 16:45:01.012358 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1289cee6bab641ada0eeb54c85047604" candidate_uuid: "d6769015f71d43edbcd5916a18c2bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" is_pre_election: true
I20260928 16:45:01.012568 6741 raft_consensus.cc:2454] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6769015f71d43edbcd5916a18c2bf00 in term 0.
I20260928 16:45:01.012930 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1289cee6bab641ada0eeb54c85047604" candidate_uuid: "d6769015f71d43edbcd5916a18c2bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d503035438ba4a33ae119110bb1e714f" is_pre_election: true
I20260928 16:45:01.013010 6812 leader_election.cc:302] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895587bd60f34bd4a311a8e6ba6f3bc8, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:01.013090 7022 raft_consensus.cc:2454] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6769015f71d43edbcd5916a18c2bf00 in term 0.
I20260928 16:45:01.013319 7104 raft_consensus.cc:2781] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:01.013391 7104 raft_consensus.cc:482] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:01.013449 7104 raft_consensus.cc:3037] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.014575 7104 raft_consensus.cc:504] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.014927 7104 leader_election.cc:288] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [CANDIDATE]: Term 1 election: Requested vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477)
I20260928 16:45:01.015406 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1289cee6bab641ada0eeb54c85047604" candidate_uuid: "d6769015f71d43edbcd5916a18c2bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d503035438ba4a33ae119110bb1e714f"
I20260928 16:45:01.015406 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1289cee6bab641ada0eeb54c85047604" candidate_uuid: "d6769015f71d43edbcd5916a18c2bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
I20260928 16:45:01.015535 6741 raft_consensus.cc:3037] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.015535 7022 raft_consensus.cc:3037] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.016738 6741 raft_consensus.cc:2454] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6769015f71d43edbcd5916a18c2bf00 in term 1.
I20260928 16:45:01.016738 7022 raft_consensus.cc:2454] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6769015f71d43edbcd5916a18c2bf00 in term 1.
I20260928 16:45:01.017011 6812 leader_election.cc:302] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d503035438ba4a33ae119110bb1e714f, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:01.017197 7104 raft_consensus.cc:2781] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:01.017403 7104 raft_consensus.cc:686] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 1 LEADER]: Becoming Leader. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Running, Role: LEADER
I20260928 16:45:01.017666 7104 consensus_queue.cc:237] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.020318 6581 catalog_manager.cc:5495] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "d6769015f71d43edbcd5916a18c2bf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } health_report { overall_health: UNKNOWN } } }
I20260928 16:45:01.082731 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.078773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.30.196.2:45703)
0928 16:45:01.079075 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:01.079078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:01.079092 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:01.079171 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:01.079173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:01.079228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:01.079333 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:01.079635 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.080283 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.081208 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.081441 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.081534 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.081805 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:01.082278 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:01.082369 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:01.082467 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:45:01.082522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
I20260928 16:45:01.085695 7022 raft_consensus.cc:1261] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Refusing update from remote peer d6769015f71d43edbcd5916a18c2bf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:01.085800 6741 raft_consensus.cc:1261] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Refusing update from remote peer d6769015f71d43edbcd5916a18c2bf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:01.086267 7104 consensus_queue.cc:1035] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:01.086537 7171 consensus_queue.cc:1035] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:01.097335 6581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:01.097517 6581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:01.100154 6581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Sending DeleteTablet for 3 replicas of tablet 1289cee6bab641ada0eeb54c85047604
I20260928 16:45:01.100767 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1289cee6bab641ada0eeb54c85047604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44006
I20260928 16:45:01.100947 7118 tablet_replica.cc:330] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: stopping tablet replica
I20260928 16:45:01.101074 7118 raft_consensus.cc:2229] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:01.101079 6720 tablet_service.cc:1514] Processing DeleteTablet for tablet 1289cee6bab641ada0eeb54c85047604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 16:45:01.101276 6719 tablet_service.cc:1514] Processing DeleteTablet for tablet 1289cee6bab641ada0eeb54c85047604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 16:45:01.101392 7118 raft_consensus.cc:2258] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:01.101439 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 1289cee6bab641ada0eeb54c85047604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:34522
I20260928 16:45:01.101457 6581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:45:01.101640 7120 tablet_replica.cc:330] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: stopping tablet replica
I20260928 16:45:01.101738 7120 raft_consensus.cc:2229] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:01.101857 7120 raft_consensus.cc:2258] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:45:01.101879 6581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:01.102042 7119 tablet_replica.cc:330] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: stopping tablet replica
I20260928 16:45:01.102164 7119 raft_consensus.cc:2229] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:01.102306 7119 raft_consensus.cc:2258] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:01.102512 7118 ts_tablet_manager.cc:1905] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:01.103596 7120 ts_tablet_manager.cc:1905] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:01.105232 7118 ts_tablet_manager.cc:1918] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:01.105306 7118 log.cc:1196] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/1289cee6bab641ada0eeb54c85047604
I20260928 16:45:01.105582 7118 ts_tablet_manager.cc:1939] T 1289cee6bab641ada0eeb54c85047604 P d6769015f71d43edbcd5916a18c2bf00: Deleting consensus metadata
I20260928 16:45:01.105723 7120 ts_tablet_manager.cc:1918] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:01.105780 7120 log.cc:1196] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/1289cee6bab641ada0eeb54c85047604
I20260928 16:45:01.105959 7120 ts_tablet_manager.cc:1939] T 1289cee6bab641ada0eeb54c85047604 P 895587bd60f34bd4a311a8e6ba6f3bc8: Deleting consensus metadata
I20260928 16:45:01.106400 6568 catalog_manager.cc:4841] TS d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703): tablet 1289cee6bab641ada0eeb54c85047604 (table test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]) successfully deleted
I20260928 16:45:01.107033 6566 catalog_manager.cc:4841] TS 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663): tablet 1289cee6bab641ada0eeb54c85047604 (table test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]) successfully deleted
W20260928 16:45:01.107638 6566 catalog_manager.cc:4808] TS 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663): delete failed for tablet 1289cee6bab641ada0eeb54c85047604 because the tablet was not found. No further retry: Not found: Tablet not found: 1289cee6bab641ada0eeb54c85047604
I20260928 16:45:01.109292 7119 ts_tablet_manager.cc:1905] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:01.110556 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 3328d995f29b41ac8ec55d96b039cdce (DEFAULT_TABLE table=test-table [id=16e4abfbaa394aa4bd6f9ffa41b9ca05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:01.110849 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3328d995f29b41ac8ec55d96b039cdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:01.110920 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 3328d995f29b41ac8ec55d96b039cdce (DEFAULT_TABLE table=test-table [id=16e4abfbaa394aa4bd6f9ffa41b9ca05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:01.111166 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3328d995f29b41ac8ec55d96b039cdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:01.112125 7119 ts_tablet_manager.cc:1918] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:01.112201 7119 log.cc:1196] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/1289cee6bab641ada0eeb54c85047604
I20260928 16:45:01.112490 7119 ts_tablet_manager.cc:1939] T 1289cee6bab641ada0eeb54c85047604 P d503035438ba4a33ae119110bb1e714f: Deleting consensus metadata
I20260928 16:45:01.113312 7096 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Bootstrap starting.
I20260928 16:45:01.113369 7097 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Bootstrap starting.
I20260928 16:45:01.114005 6719 tablet_service.cc:1467] Processing CreateTablet for tablet 3328d995f29b41ac8ec55d96b039cdce (DEFAULT_TABLE table=test-table [id=16e4abfbaa394aa4bd6f9ffa41b9ca05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:01.114213 7097 tablet_bootstrap.cc:654] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:01.114300 7096 tablet_bootstrap.cc:654] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:01.114311 6719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3328d995f29b41ac8ec55d96b039cdce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:01.115897 7096 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: No bootstrap required, opened a new log
I20260928 16:45:01.115988 7096 ts_tablet_manager.cc:1397] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 16:45:01.116442 7096 raft_consensus.cc:348] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.116560 7096 raft_consensus.cc:374] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:01.116590 7096 raft_consensus.cc:729] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769015f71d43edbcd5916a18c2bf00, State: Initialized, Role: FOLLOWER
I20260928 16:45:01.116707 7096 consensus_queue.cc:260] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.116813 7098 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: Bootstrap starting.
I20260928 16:45:01.116942 7096 ts_tablet_manager.cc:1428] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:45:01.117147 6566 catalog_manager.cc:4841] TS d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477): tablet 1289cee6bab641ada0eeb54c85047604 (table test-table [id=118eb3b6bc504057b9d7c7a23c31e7d1]) successfully deleted
I20260928 16:45:01.117669 7098 tablet_bootstrap.cc:654] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:01.118896 7098 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: No bootstrap required, opened a new log
I20260928 16:45:01.118980 7098 ts_tablet_manager.cc:1397] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:01.119377 7098 raft_consensus.cc:348] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.119498 7098 raft_consensus.cc:374] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:01.119530 7098 raft_consensus.cc:729] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895587bd60f34bd4a311a8e6ba6f3bc8, State: Initialized, Role: FOLLOWER
I20260928 16:45:01.119665 7097 tablet_bootstrap.cc:492] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: No bootstrap required, opened a new log
I20260928 16:45:01.119649 7098 consensus_queue.cc:260] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.119776 7097 ts_tablet_manager.cc:1397] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 16:45:01.119930 7098 ts_tablet_manager.cc:1428] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:45:01.120222 7097 raft_consensus.cc:348] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.120330 7097 raft_consensus.cc:374] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:01.120388 7097 raft_consensus.cc:729] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Initialized, Role: FOLLOWER
I20260928 16:45:01.120518 7097 consensus_queue.cc:260] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.120858 7097 ts_tablet_manager.cc:1428] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:45:01.242311 7106 raft_consensus.cc:482] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:01.242458 7106 raft_consensus.cc:504] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.242992 7106 leader_election.cc:288] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703)
I20260928 16:45:01.243474 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3328d995f29b41ac8ec55d96b039cdce" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" is_pre_election: true
I20260928 16:45:01.243515 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3328d995f29b41ac8ec55d96b039cdce" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00" is_pre_election: true
I20260928 16:45:01.243634 6741 raft_consensus.cc:2454] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:01.243667 6882 raft_consensus.cc:2454] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 0.
I20260928 16:45:01.244020 6955 leader_election.cc:302] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d503035438ba4a33ae119110bb1e714f, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:01.244213 7106 raft_consensus.cc:2781] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:01.244264 7106 raft_consensus.cc:482] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:01.244305 7106 raft_consensus.cc:3037] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.245587 7106 raft_consensus.cc:504] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.245894 7106 leader_election.cc:288] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Requested vote from peers 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663), d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703)
I20260928 16:45:01.246282 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3328d995f29b41ac8ec55d96b039cdce" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6769015f71d43edbcd5916a18c2bf00"
I20260928 16:45:01.246384 6882 raft_consensus.cc:3037] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.247385 6882 raft_consensus.cc:2454] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:01.247690 6955 leader_election.cc:302] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d503035438ba4a33ae119110bb1e714f, d6769015f71d43edbcd5916a18c2bf00; no voters:
I20260928 16:45:01.247876 7106 raft_consensus.cc:2781] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:01.248004 7106 raft_consensus.cc:686] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Becoming Leader. State: Replica: d503035438ba4a33ae119110bb1e714f, State: Running, Role: LEADER
I20260928 16:45:01.248143 7106 consensus_queue.cc:237] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } }
I20260928 16:45:01.250969 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3328d995f29b41ac8ec55d96b039cdce" candidate_uuid: "d503035438ba4a33ae119110bb1e714f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8"
I20260928 16:45:01.251139 6741 raft_consensus.cc:3037] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:01.251515 6581 catalog_manager.cc:5495] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f reported cstate change: term changed from 0 to 1, leader changed from <none> to d503035438ba4a33ae119110bb1e714f (127.30.196.3). New cstate: current_term: 1 leader_uuid: "d503035438ba4a33ae119110bb1e714f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d503035438ba4a33ae119110bb1e714f" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 33477 } health_report { overall_health: HEALTHY } } }
I20260928 16:45:01.252389 6741 raft_consensus.cc:2454] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d503035438ba4a33ae119110bb1e714f in term 1.
I20260928 16:45:01.285290 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.281915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.30.196.3:33477)
0928 16:45:01.282032 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:01.282036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:01.282148 (+ 112us) server_negotiation.cc:407] Connection header received
0928 16:45:01.282264 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:01.282266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:01.282312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:01.282375 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:01.282611 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.283133 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.284111 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:01.284357 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:01.284485 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:01.284769 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:01.284847 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:01.284950 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:01.285106 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:45:01.285167 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 16:45:01.291863 6881 raft_consensus.cc:1261] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:01.292305 7182 consensus_queue.cc:1035] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6769015f71d43edbcd5916a18c2bf00" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 45703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:01.292441 6741 raft_consensus.cc:1261] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Refusing update from remote peer d503035438ba4a33ae119110bb1e714f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:01.292881 7182 consensus_queue.cc:1035] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [LEADER]: Connected to new peer: Peer: permanent_uuid: "895587bd60f34bd4a311a8e6ba6f3bc8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:01.328018 6583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:01.328197 6583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:01.330544 6583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da5895f8e74c49929dec6285976bbd54: Sending DeleteTablet for 3 replicas of tablet 3328d995f29b41ac8ec55d96b039cdce
I20260928 16:45:01.331211 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 3328d995f29b41ac8ec55d96b039cdce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44006
I20260928 16:45:01.331437 7118 tablet_replica.cc:330] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: stopping tablet replica
I20260928 16:45:01.331434 6719 tablet_service.cc:1514] Processing DeleteTablet for tablet 3328d995f29b41ac8ec55d96b039cdce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 16:45:01.331596 7118 raft_consensus.cc:2229] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:01.331626 7120 tablet_replica.cc:330] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8: stopping tablet replica
I20260928 16:45:01.331781 7120 raft_consensus.cc:2229] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:01.331799 7118 raft_consensus.cc:2258] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:01.331907 7120 raft_consensus.cc:2258] T 3328d995f29b41ac8ec55d96b039cdce P 895587bd60f34bd4a311a8e6ba6f3bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:01.331941 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6655
I20260928 16:45:01.334343 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 3328d995f29b41ac8ec55d96b039cdce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:01 UTC) from {username='oryx', principal='oryx/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:34522
I20260928 16:45:01.334559 7119 tablet_replica.cc:330] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: stopping tablet replica
I20260928 16:45:01.334575 7118 ts_tablet_manager.cc:1905] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:01.334723 7119 raft_consensus.cc:2229] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:01.334964 7119 raft_consensus.cc:2258] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:01.335564 7119 ts_tablet_manager.cc:1905] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:01.337749 7119 ts_tablet_manager.cc:1918] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:45:01.337858 7119 log.cc:1196] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/3328d995f29b41ac8ec55d96b039cdce
I20260928 16:45:01.338033 7118 ts_tablet_manager.cc:1918] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:45:01.338137 7118 log.cc:1196] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/3328d995f29b41ac8ec55d96b039cdce
I20260928 16:45:01.338150 7119 ts_tablet_manager.cc:1939] T 3328d995f29b41ac8ec55d96b039cdce P d503035438ba4a33ae119110bb1e714f: Deleting consensus metadata
I20260928 16:45:01.338419 7118 ts_tablet_manager.cc:1939] T 3328d995f29b41ac8ec55d96b039cdce P d6769015f71d43edbcd5916a18c2bf00: Deleting consensus metadata
I20260928 16:45:01.338958 6566 catalog_manager.cc:4841] TS d503035438ba4a33ae119110bb1e714f (127.30.196.3:33477): tablet 3328d995f29b41ac8ec55d96b039cdce (table test-table [id=16e4abfbaa394aa4bd6f9ffa41b9ca05]) successfully deleted
I20260928 16:45:01.339221 6568 catalog_manager.cc:4841] TS d6769015f71d43edbcd5916a18c2bf00 (127.30.196.2:45703): tablet 3328d995f29b41ac8ec55d96b039cdce (table test-table [id=16e4abfbaa394aa4bd6f9ffa41b9ca05]) successfully deleted
W20260928 16:45:01.341622 6566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:40663 (error 108)
I20260928 16:45:01.341967 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6796
I20260928 16:45:01.344375 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:01.343967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40663 (local address 127.0.0.1:59684)
0928 16:45:01.344197 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:01.344264 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
W20260928 16:45:01.344640 6566 catalog_manager.cc:4584] TS 895587bd60f34bd4a311a8e6ba6f3bc8 (127.30.196.1:40663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3328d995f29b41ac8ec55d96b039cdce: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40663: connect: Connection refused (error 111)
I20260928 16:45:01.350167 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6937
I20260928 16:45:01.357697 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 6550
2026-09-28T16:45:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3692 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:45:01 dist-test-slave-5rg1 krb5kdc[7195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:45:01 dist-test-slave-5rg1 krb5kdc[7195](info): set up 2 sockets
Sep 28 16:45:01 dist-test-slave-5rg1 krb5kdc[7195](info): commencing operation
krb5kdc: starting...
W20260928 16:45:02.436826 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:45:02 dist-test-slave-5rg1 krb5kdc[7195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:45:02.571790 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37167
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:37167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:02.695874 7208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:02.696164 7208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:02.696205 7208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:02.700381 7208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:02.700446 7208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:02.700467 7208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:02.700484 7208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:02.700531 7208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:02.705942 7208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:37167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:02.707087 7208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:02.708278 7208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:02.717072 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:02.717090 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:02.717139 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:02.717509 7208 server_base.cc:1034] running on GCE node
I20260928 16:45:02.718137 7208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:02.719360 7208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:02.720582 7208 hybrid_clock.cc:648] HybridClock initialized: now 1790613902720538 us; error 65 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T16:45:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1380 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:45:02 dist-test-slave-5rg1 krb5kdc[7221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:45:02 dist-test-slave-5rg1 krb5kdc[7221](info): set up 2 sockets
Sep 28 16:45:02 dist-test-slave-5rg1 krb5kdc[7221](info): commencing operation
krb5kdc: starting...
W20260928 16:45:03.796557 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:45:03 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:45:03.923839 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46759
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:04.037865 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:04.038174 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:04.038218 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:04.042459 7234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:04.042532 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:04.042553 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:04.042570 7234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:04.042586 7234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:04.048247 7234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:04.049458 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:04.050518 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:45:04.059592 7234 server_base.cc:1034] running on GCE node
W20260928 16:45:04.059525 7240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.059475 7239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.059475 7242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:04.060525 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:04.061653 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:04.062817 7234 hybrid_clock.cc:648] HybridClock initialized: now 1790613904062797 us; error 33 us; skew 500 ppm
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:04.065907 7234 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:45:04.066859 7234 webserver.cc:460] Webserver started at http://127.30.196.62:35863/ using document root <none> and password file <none>
I20260928 16:45:04.067440 7234 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:04.067503 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:04.067783 7234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:04.069867 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "024b81320349458f9276374eb2a28fc6"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "d5b4c12caca3d024fa6c608a88b898d6"
server_key_iv: "44fa85b16c23e955bd05f423484f118f"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.070454 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "024b81320349458f9276374eb2a28fc6"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "d5b4c12caca3d024fa6c608a88b898d6"
server_key_iv: "44fa85b16c23e955bd05f423484f118f"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.074292 7234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:45:04.076687 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.078086 7234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:04.078253 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "024b81320349458f9276374eb2a28fc6"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "d5b4c12caca3d024fa6c608a88b898d6"
server_key_iv: "44fa85b16c23e955bd05f423484f118f"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.078375 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:04.091053 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:04.094274 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:04.094542 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:04.102061 7234 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:46759
I20260928 16:45:04.102078 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:46759 every 8 connection(s)
I20260928 16:45:04.103092 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:04.105899 7302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:04.110945 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7234
I20260928 16:45:04.111091 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:45:04.111392 31504 external_mini_cluster.cc:1428] Setting key ff9eeb068689fa0ed0464aa0a292b2fc
I20260928 16:45:04.111886 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Bootstrap starting.
I20260928 16:45:04.114451 7302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:04.115458 7302 log.cc:824] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:04.117883 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: No bootstrap required, opened a new log
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:04.124177 7302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:04.124511 7302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:04.124598 7302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024b81320349458f9276374eb2a28fc6, State: Initialized, Role: FOLLOWER
I20260928 16:45:04.125133 7302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:04.125279 7302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:04.125373 7302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:04.125473 7302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:04.126435 7302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:04.126799 7302 leader_election.cc:302] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 024b81320349458f9276374eb2a28fc6; no voters:
I20260928 16:45:04.127132 7302 leader_election.cc:288] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:45:04.127265 7307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:04.127507 7307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 024b81320349458f9276374eb2a28fc6, State: Running, Role: LEADER
I20260928 16:45:04.127875 7307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:04.128473 7302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:04.129200 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024b81320349458f9276374eb2a28fc6. Latest consensus state: current_term: 1 leader_uuid: "024b81320349458f9276374eb2a28fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } } }
I20260928 16:45:04.129311 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:04.129666 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "024b81320349458f9276374eb2a28fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } } }
I20260928 16:45:04.129760 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:04.129827 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.112973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.30.196.62:46759)
0928 16:45:04.113458 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:04.113475 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:04.113516 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:45:04.114167 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:04.114185 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:04.114559 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:04.115003 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:04.116297 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.117794 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.119086 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.119915 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.120086 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.123639 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:04.123651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:04.123716 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:04.123753 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:04.125714 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.126224 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.126231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.126238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.126370 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.126688 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.126693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.126695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.127265 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:04.128282 (+ 1017us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:04.128443 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 16:45:04.128770 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":160,"threads_started":1}
W20260928 16:45:04.133704 7322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:45:04.133792 7322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:45:04.133852 7313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:04.134744 7313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:04.140630 7313 catalog_manager.cc:1347] Generated new cluster ID: 5e0a8ec5c0414f34ae3c21746c829bea
I20260928 16:45:04.140713 7313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:04.168411 7313 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:45:04.169747 7313 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:04.177322 7313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Generated new TSK 0
I20260928 16:45:04.178148 7313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:45:04.215111 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:04.334332 7329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:04.334622 7329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:04.334668 7329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:04.338869 7329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:04.338948 7329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:04.339011 7329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:45:04.344640 7329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:04.345697 7329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:04.346694 7329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:04.356701 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.356698 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:04.356752 7329 server_base.cc:1034] running on GCE node
W20260928 16:45:04.356699 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:04.357357 7329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:04.358006 7329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:04.359210 7329 hybrid_clock.cc:648] HybridClock initialized: now 1790613904359192 us; error 28 us; skew 500 ppm
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:04.362622 7329 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:45:04.363629 7329 webserver.cc:460] Webserver started at http://127.30.196.1:33797/ using document root <none> and password file <none>
I20260928 16:45:04.364238 7329 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:04.364331 7329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:04.364571 7329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:04.367087 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "4ebaa507209542d290258bb2f5369523"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "a3ce9364249a8e4b35399e9a3ecea127"
server_key_iv: "57a28734905663711f0cdf3a26c81d5b"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.367736 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "4ebaa507209542d290258bb2f5369523"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "a3ce9364249a8e4b35399e9a3ecea127"
server_key_iv: "57a28734905663711f0cdf3a26c81d5b"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.371656 7329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:45:04.374454 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.375828 7329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:04.375969 7329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "4ebaa507209542d290258bb2f5369523"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "a3ce9364249a8e4b35399e9a3ecea127"
server_key_iv: "57a28734905663711f0cdf3a26c81d5b"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.376101 7329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:04.388096 7329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:04.392243 7329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:04.392498 7329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:04.393164 7329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:04.394076 7329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:04.394129 7329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.394205 7329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:04.394241 7329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.403422 7329 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:34031
I20260928 16:45:04.403456 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:34031 every 8 connection(s)
I20260928 16:45:04.404497 7329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:04.412371 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7329
I20260928 16:45:04.412534 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:45:04.412878 31504 external_mini_cluster.cc:1428] Setting key 89e4b94e0eb0a4611f13b4b014e48b0d
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:04.419858 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.406452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:41127 (local address 127.30.196.62:46759)
0928 16:45:04.406600 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:04.406604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:04.407519 (+ 915us) server_negotiation.cc:407] Connection header received
0928 16:45:04.409032 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:04.409036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:04.409095 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:04.409210 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:04.410388 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.410933 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.412173 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.412410 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.412549 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.415411 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:04.415414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:04.415416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:04.415440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:04.417251 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.417967 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.417970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.417971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.418111 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.418566 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.418569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.418570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.418746 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:04.418822 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:04.419541 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 16:45:04.419659 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:45:04.421166 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.406842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:41127)
0928 16:45:04.407333 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:04.407376 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:04.408795 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:04.409384 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:04.409391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:04.409751 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:04.410239 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.410251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.411071 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.411079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.412046 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.412055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.412633 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.412640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.412865 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.413600 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:04.413618 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:04.415236 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:04.417424 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.417433 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.417453 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.417862 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.418217 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.418223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.418227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.418434 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.418860 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:04.418869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:04.419240 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 16:45:04.419485 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 16:45:04.419930 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260928 16:45:04.422519 7459 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46759
I20260928 16:45:04.422783 7459 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:04.423295 7459 heartbeater.cc:511] Master 127.30.196.62:46759 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:45:04.425024 7266 ts_manager.cc:194] Registered new tserver with Master: 4ebaa507209542d290258bb2f5369523 (127.30.196.1:34031)
I20260928 16:45:04.426322 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:41127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:45:04.443356 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:04.559015 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:04.559319 7464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:04.559394 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:04.563881 7464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:04.563993 7464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:04.564083 7464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:45:04.569442 7464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:04.570523 7464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:04.571547 7464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:04.581432 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.581506 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.581467 7473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:04.581594 7464 server_base.cc:1034] running on GCE node
I20260928 16:45:04.582175 7464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:04.582747 7464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:04.583904 7464 hybrid_clock.cc:648] HybridClock initialized: now 1790613904583891 us; error 31 us; skew 500 ppm
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:04.586621 7464 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:45:04.587452 7464 webserver.cc:460] Webserver started at http://127.30.196.2:46555/ using document root <none> and password file <none>
I20260928 16:45:04.588056 7464 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:04.588120 7464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:04.588295 7464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:04.590358 7464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "41d923eb8ab94fad97b804ef751e4947"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "042eeca48f0df9576257d71a39a0d5de"
server_key_iv: "7a578ec72e2976899f5c50f5e74c8dd4"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.590880 7464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "41d923eb8ab94fad97b804ef751e4947"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "042eeca48f0df9576257d71a39a0d5de"
server_key_iv: "7a578ec72e2976899f5c50f5e74c8dd4"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.594758 7464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:45:04.597569 7480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.599081 7464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:45:04.599205 7464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "41d923eb8ab94fad97b804ef751e4947"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "042eeca48f0df9576257d71a39a0d5de"
server_key_iv: "7a578ec72e2976899f5c50f5e74c8dd4"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.599370 7464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:04.613498 7464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:04.617058 7464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:04.617272 7464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:04.617893 7464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:04.618731 7464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:04.618778 7464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.618850 7464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:04.618887 7464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.627899 7464 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:33091
I20260928 16:45:04.627951 7593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:33091 every 8 connection(s)
I20260928 16:45:04.629149 7464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:04.630462 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7464
I20260928 16:45:04.630614 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:45:04.630905 31504 external_mini_cluster.cc:1428] Setting key 2e04c68ea527d37d487dfd30138afff4
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:45:04.646209 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.631362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:52829 (local address 127.30.196.62:46759)
0928 16:45:04.631528 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:04.631533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:04.632620 (+ 1087us) server_negotiation.cc:407] Connection header received
0928 16:45:04.634739 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:04.634743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:04.634808 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:04.634891 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:04.636309 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.637181 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.638563 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.639012 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.639171 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.642104 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:04.642110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:04.642115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:04.642165 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:04.643601 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.644255 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.644258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.644260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.644378 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.644762 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.644764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.644766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.645105 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:04.645227 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:04.645788 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 16:45:04.645950 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:45:04.646919 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.631893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:52829)
0928 16:45:04.632442 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:04.632507 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:04.634445 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:04.635092 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:04.635099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:04.635499 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:04.636140 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.636156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.637369 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.637372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.638406 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.638418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.639214 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.639217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.639392 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.640140 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:04.640157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:04.641899 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:04.643764 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.643769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.643784 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.644052 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.644477 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.644479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.644480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.644628 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.645231 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:04.645237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:04.645505 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 16:45:04.645674 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:45:04.645956 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":111,"threads_started":1}
I20260928 16:45:04.648361 7594 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46759
I20260928 16:45:04.648624 7594 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:04.649161 7594 heartbeater.cc:511] Master 127.30.196.62:46759 requested a full tablet report, sending...
I20260928 16:45:04.650243 7266 ts_manager.cc:194] Registered new tserver with Master: 41d923eb8ab94fad97b804ef751e4947 (127.30.196.2:33091)
I20260928 16:45:04.650854 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:52829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:45:04.664268 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:04.777249 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:04.777670 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:04.777737 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:04.782199 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:04.782265 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:04.782327 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:45:04.787977 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:04.789070 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:04.790130 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:04.800122 7606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.800122 7605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:04.800159 7608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:04.800295 7599 server_base.cc:1034] running on GCE node
I20260928 16:45:04.800858 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:04.801597 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:04.802758 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790613904802731 us; error 43 us; skew 500 ppm
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:04.805655 7599 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:45:04.806535 7599 webserver.cc:460] Webserver started at http://127.30.196.3:35421/ using document root <none> and password file <none>
I20260928 16:45:04.807073 7599 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:04.807133 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:04.807300 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:04.809381 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "0e45ebccce794674a46e9f144f61d0ff"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "b4f30a94c486e9d941a5cb39e8424790"
server_key_iv: "7e232a97b928d1405ff141f65506b069"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.809947 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "0e45ebccce794674a46e9f144f61d0ff"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "b4f30a94c486e9d941a5cb39e8424790"
server_key_iv: "7e232a97b928d1405ff141f65506b069"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.813756 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:45:04.816241 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.817543 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:04.817698 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "0e45ebccce794674a46e9f144f61d0ff"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "b4f30a94c486e9d941a5cb39e8424790"
server_key_iv: "7e232a97b928d1405ff141f65506b069"
server_key_version: "encryptionkey@0"
I20260928 16:45:04.817814 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:04.836721 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:04.840265 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:04.840507 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:04.841228 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:04.842098 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:04.842171 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.842247 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:04.842298 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:04.851799 7599 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:38401
I20260928 16:45:04.851821 7728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:38401 every 8 connection(s)
I20260928 16:45:04.852702 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:04.861542 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7599
I20260928 16:45:04.861754 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:45:04.862128 31504 external_mini_cluster.cc:1428] Setting key 9ed920beeeacc3f36b8fe113c2686dba
Sep 28 16:45:04 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613904, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:04.867362 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.854580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:41935 (local address 127.30.196.62:46759)
0928 16:45:04.854774 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:04.854779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:04.855697 (+ 918us) server_negotiation.cc:407] Connection header received
0928 16:45:04.857169 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:04.857173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:04.857224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:04.857307 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:04.858640 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.859197 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.860502 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.860716 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.860800 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.863927 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:04.863931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:04.863934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:04.863965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:04.865414 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.865924 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.865927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.865929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.866010 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:04.866343 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:04.866345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:04.866346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:04.866528 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:04.866601 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:04.867076 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 16:45:04.867238 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:45:04.868271 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:04.854983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:41935)
0928 16:45:04.855492 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:04.855539 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:04.856891 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:04.857530 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:04.857537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:04.857919 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:04.858466 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.858482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.859345 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.859347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.860366 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:04.860376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:04.860906 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:04.860910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:04.861068 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:04.861869 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:04.861902 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:04.863765 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:04.865533 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.865538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.865554 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.865818 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.866105 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:04.866108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:04.866109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:04.866251 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:04.866619 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:04.866624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:04.866863 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 16:45:04.867015 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 16:45:04.867308 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
I20260928 16:45:04.869489 7729 heartbeater.cc:347] Connected to a master server at 127.30.196.62:46759
I20260928 16:45:04.869719 7729 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:04.870198 7729 heartbeater.cc:511] Master 127.30.196.62:46759 requested a full tablet report, sending...
I20260928 16:45:04.871254 7266 ts_manager.cc:194] Registered new tserver with Master: 0e45ebccce794674a46e9f144f61d0ff (127.30.196.3:38401)
I20260928 16:45:04.871940 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:41935
I20260928 16:45:04.876537 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
Entry for principal oryx/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab.
WARNING: no policy specified for oryx/127.30.196.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
Entry for principal oryx/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab.
WARNING: no policy specified for oryx/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.196.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
Entry for principal oryx/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab.
I20260928 16:45:04.958657 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7329
I20260928 16:45:04.966835 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7464
I20260928 16:45:04.972716 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7599
I20260928 16:45:04.978514 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7234
I20260928 16:45:04.985189 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46759
--webserver_interface=127.30.196.62
--webserver_port=35863
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:05.097527 7738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:05.097842 7738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:05.097925 7738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:05.101953 7738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:05.102046 7738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:05.102098 7738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:05.102159 7738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:05.102197 7738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:05.107496 7738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=35863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:05.108669 7738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:05.109745 7738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:05.118562 7743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.118695 7738 server_base.cc:1034] running on GCE node
W20260928 16:45:05.118589 7744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.118600 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.119465 7738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:05.120682 7738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:05.121876 7738 hybrid_clock.cc:648] HybridClock initialized: now 1790613905121862 us; error 38 us; skew 500 ppm
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613905, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:05.124691 7738 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:45:05.125599 7738 webserver.cc:460] Webserver started at http://127.30.196.62:35863/ using document root <none> and password file <none>
I20260928 16:45:05.126200 7738 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:05.126330 7738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:05.130250 7738 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:05.132589 7753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.133859 7738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:05.134003 7738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "024b81320349458f9276374eb2a28fc6"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "d5b4c12caca3d024fa6c608a88b898d6"
server_key_iv: "44fa85b16c23e955bd05f423484f118f"
server_key_version: "encryptionkey@0"
I20260928 16:45:05.134567 7738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:05.151297 7738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:05.154922 7738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:05.155153 7738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:05.162819 7738 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:46759
I20260928 16:45:05.162889 7806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:46759 every 8 connection(s)
I20260928 16:45:05.164057 7738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:05.168331 7807 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:45:05.169986 7807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Bootstrap starting.
I20260928 16:45:05.171859 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7738
I20260928 16:45:05.172304 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:34031
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:46759
--webserver_port=33797
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:45:05.187561 7807 log.cc:824] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:05.194365 7807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:45:05.194778 7807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Bootstrap complete.
I20260928 16:45:05.199234 7807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:05.199718 7807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024b81320349458f9276374eb2a28fc6, State: Initialized, Role: FOLLOWER
I20260928 16:45:05.200258 7807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:05.200376 7807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:05.200462 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:05.200564 7807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:45:05.202579 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:05.202925 7807 leader_election.cc:302] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 024b81320349458f9276374eb2a28fc6; no voters:
I20260928 16:45:05.203291 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:45:05.203384 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:45:05.203608 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [term 2 LEADER]: Becoming Leader. State: Replica: 024b81320349458f9276374eb2a28fc6, State: Running, Role: LEADER
I20260928 16:45:05.203935 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } }
I20260928 16:45:05.204584 7807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:05.204977 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024b81320349458f9276374eb2a28fc6. Latest consensus state: current_term: 2 leader_uuid: "024b81320349458f9276374eb2a28fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } } }
I20260928 16:45:05.205096 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:05.205495 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "024b81320349458f9276374eb2a28fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024b81320349458f9276374eb2a28fc6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46759 } } }
I20260928 16:45:05.205667 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:05.205569 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:05.209360 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:05.210009 7815 catalog_manager.cc:1259] Loaded cluster ID: 5e0a8ec5c0414f34ae3c21746c829bea
I20260928 16:45:05.210074 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:05.212514 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:05.213232 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 024b81320349458f9276374eb2a28fc6: Loaded TSK: 0
I20260928 16:45:05.214470 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:45:05.291931 7809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:05.292266 7809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:05.292348 7809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:05.296609 7809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:05.296710 7809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:05.296803 7809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:45:05.302201 7809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.1.keytab
--principal=oryx/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:34031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=33797
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:05.303280 7809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:05.304360 7809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:05.314210 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.314210 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.314787 7836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.314786 7809 server_base.cc:1034] running on GCE node
I20260928 16:45:05.315219 7809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:05.316026 7809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:05.317211 7809 hybrid_clock.cc:648] HybridClock initialized: now 1790613905317189 us; error 40 us; skew 500 ppm
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613905, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:05.320189 7809 init.cc:377] Logged in from keytab as oryx/127.30.196.1@KRBTEST.COM (short username oryx)
I20260928 16:45:05.321156 7809 webserver.cc:460] Webserver started at http://127.30.196.1:33797/ using document root <none> and password file <none>
I20260928 16:45:05.321753 7809 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:05.321870 7809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:05.326022 7809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 16:45:05.328332 7843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.329744 7809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:05.329887 7809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "4ebaa507209542d290258bb2f5369523"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "a3ce9364249a8e4b35399e9a3ecea127"
server_key_iv: "57a28734905663711f0cdf3a26c81d5b"
server_key_version: "encryptionkey@0"
I20260928 16:45:05.330451 7809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:05.358847 7809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:05.362461 7809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:05.362684 7809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:05.363399 7809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:05.364279 7809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:05.364351 7809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.364424 7809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:05.364472 7809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.373778 7809 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:34031
I20260928 16:45:05.373822 7957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:34031 every 8 connection(s)
I20260928 16:45:05.374832 7809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:05.379837 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7809
I20260928 16:45:05.380370 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:33091
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:46759
--webserver_port=46555
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:45:05.389539 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.377189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:36393 (local address 127.30.196.62:46759)
0928 16:45:05.377689 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.377704 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.378117 (+ 413us) server_negotiation.cc:407] Connection header received
0928 16:45:05.379752 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.379768 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.380277 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.380650 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.381921 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.382920 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.384220 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.384733 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.384924 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.388724 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:36393: BlockingRecv error: recv got EOF from 127.30.196.1:36393 (error 108)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":137,"threads_started":1}
I20260928 16:45:05.389547 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.377252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:36393)
0928 16:45:05.377930 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.377973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.379247 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.380832 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.380839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.381262 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.381722 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.381737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.383085 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.383089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.384080 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.384091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.384917 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.384921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.385127 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.385805 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.385822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.388479 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":135,"threads_started":1}
W20260928 16:45:05.390372 7958 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:46759 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.196.62:46759: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.397260 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.390951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:52193)
0928 16:45:05.391140 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.391154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.391236 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.391738 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.391741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.391979 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.392105 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.392113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.393119 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.393125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.393952 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.393965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.394447 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.394450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.394546 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.395165 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.395184 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.397118 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 16:45:05.397650 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.391210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:52193 (local address 127.30.196.62:46759)
0928 16:45:05.391387 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.391392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.391407 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:05.391472 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.391475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.391535 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.391631 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.392241 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.393001 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.394100 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.394326 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.394440 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.397264 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:52193: BlockingRecv error: recv got EOF from 127.30.196.1:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.403343 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.397852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:48661)
0928 16:45:05.397976 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.397988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.398065 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.398393 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.398396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.398567 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.398661 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.398670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.399386 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.399389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.400290 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.400302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.400751 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.400754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.400840 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.401295 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.401311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.403195 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:45:05.403432 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.398109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:48661 (local address 127.30.196.62:46759)
0928 16:45:05.398206 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.398209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.398219 (+ 10us) server_negotiation.cc:407] Connection header received
0928 16:45:05.398249 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.398251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.398285 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.398342 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.398772 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.399284 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.400430 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.400648 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.400728 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.403316 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:48661: BlockingRecv error: recv got EOF from 127.30.196.1:48661 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 16:45:05.403790 7958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:45:05.498994 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:05.499317 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:05.499393 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:05.503832 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:05.503950 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:05.504045 7962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:45:05.510098 7962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.2.keytab
--principal=oryx/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:33091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=46555
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:05.511250 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:05.512324 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:05.523011 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.523011 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.523015 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.523787 7962 server_base.cc:1034] running on GCE node
I20260928 16:45:05.524331 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:05.525256 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:05.526506 7962 hybrid_clock.cc:648] HybridClock initialized: now 1790613905526459 us; error 36 us; skew 500 ppm
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613905, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:05.529582 7962 init.cc:377] Logged in from keytab as oryx/127.30.196.2@KRBTEST.COM (short username oryx)
I20260928 16:45:05.530552 7962 webserver.cc:460] Webserver started at http://127.30.196.2:46555/ using document root <none> and password file <none>
I20260928 16:45:05.531139 7962 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:05.531241 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:05.535789 7962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:05.538028 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.539624 7962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:45:05.539825 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "41d923eb8ab94fad97b804ef751e4947"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "042eeca48f0df9576257d71a39a0d5de"
server_key_iv: "7a578ec72e2976899f5c50f5e74c8dd4"
server_key_version: "encryptionkey@0"
I20260928 16:45:05.540385 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:05.551731 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:05.555193 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:05.555413 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:05.556100 7962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:05.556962 7962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:05.557009 7962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.557050 7962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:05.557065 7962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.566121 7962 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:33091
I20260928 16:45:05.566262 8091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:33091 every 8 connection(s)
I20260928 16:45:05.567180 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:05.567648 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 7962
I20260928 16:45:05.568174 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:38401
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:46759
--webserver_port=35421
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:45:05.581735 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.569362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:54413 (local address 127.30.196.62:46759)
0928 16:45:05.569548 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.569553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.570438 (+ 885us) server_negotiation.cc:407] Connection header received
0928 16:45:05.572477 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.572481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.572553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.572662 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.574444 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.575122 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.577083 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.577302 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.577400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.581627 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:54413: BlockingRecv error: recv got EOF from 127.30.196.2:54413 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:45:05.582480 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.569774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:54413)
0928 16:45:05.570265 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.570316 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.572175 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.572892 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.572903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.573475 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.574231 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.574257 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.575274 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.575280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.576923 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.576938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.577490 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.577497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.577740 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.578436 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.578462 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.581481 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":165,"threads_started":1}
W20260928 16:45:05.583035 8092 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:46759 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.196.62:46759: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.589180 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.583513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:55275)
0928 16:45:05.583682 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.583698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.583781 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.584077 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.584080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.584250 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.584361 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.584367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.585133 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.585137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.586017 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.586026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.586450 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.586453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.586555 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.587031 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.587046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.589052 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 16:45:05.589469 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.583633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:55275 (local address 127.30.196.62:46759)
0928 16:45:05.583762 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.583765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.583778 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:45:05.583916 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.583919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.583962 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.584046 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.584487 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.584990 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.586135 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.586337 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.586415 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.589199 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:55275: BlockingRecv error: recv got EOF from 127.30.196.2:55275 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.595068 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.589727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:57965)
0928 16:45:05.589849 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.589865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.589941 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.590233 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.590236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.590418 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.590506 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.590512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.591266 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.591269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.592135 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.592146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.592592 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.592595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.592681 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.593151 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.593167 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.594951 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:45:05.595183 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.589945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:57965 (local address 127.30.196.62:46759)
0928 16:45:05.590055 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.590058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.590069 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:45:05.590099 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.590101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.590134 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.590196 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.590665 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.591161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.592273 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.592476 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.592567 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.595069 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:57965: BlockingRecv error: recv got EOF from 127.30.196.2:57965 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 16:45:05.595368 8092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:45:05.690451 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:05.690769 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:05.690848 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:05.695114 8094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:05.695200 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:05.695291 8094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:45:05.700722 8094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/krb5kdc/oryx_127.30.196.3.keytab
--principal=oryx/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:38401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=35421
--tserver_master_addrs=127.30.196.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:05.701848 8094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:05.702895 8094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:05.713140 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:05.713153 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.713243 8094 server_base.cc:1034] running on GCE node
W20260928 16:45:05.713200 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:05.713812 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:05.714479 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:05.715723 8094 hybrid_clock.cc:648] HybridClock initialized: now 1790613905715681 us; error 45 us; skew 500 ppm
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613905, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:05.718425 8094 init.cc:377] Logged in from keytab as oryx/127.30.196.3@KRBTEST.COM (short username oryx)
I20260928 16:45:05.719340 8094 webserver.cc:460] Webserver started at http://127.30.196.3:35421/ using document root <none> and password file <none>
I20260928 16:45:05.719980 8094 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:05.720093 8094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:05.724054 8094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:45:05.726217 8110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.727485 8094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:05.727646 8094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "0e45ebccce794674a46e9f144f61d0ff"
format_stamp: "Formatted at 2026-09-28 16:45:04 on dist-test-slave-5rg1"
server_key: "b4f30a94c486e9d941a5cb39e8424790"
server_key_iv: "7e232a97b928d1405ff141f65506b069"
server_key_version: "encryptionkey@0"
I20260928 16:45:05.728300 8094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:05.755554 8094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:05.759306 8094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:05.759534 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:05.760190 8094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:05.761052 8094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:05.761128 8094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.761228 8094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:05.761281 8094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:05.771157 8094 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:38401
I20260928 16:45:05.771201 8224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:38401 every 8 connection(s)
I20260928 16:45:05.772300 8094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:05.776468 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8094
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:45:05.784541 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.774333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:42103 (local address 127.30.196.62:46759)
0928 16:45:05.774484 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.774489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.775244 (+ 755us) server_negotiation.cc:407] Connection header received
0928 16:45:05.776961 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.776965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.777037 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.777142 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.778782 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.779320 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.780580 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.780803 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.780930 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.784414 (+ 3484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:42103: BlockingRecv error: recv got EOF from 127.30.196.3:42103 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:45:05.785238 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.774583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:42103)
0928 16:45:05.775077 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.775123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.776668 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.777448 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.777458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.777969 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.778628 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.778644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.779469 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.779472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.780449 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.780460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.780965 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.780969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.781193 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.781749 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.781766 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.784310 (+ 2544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
W20260928 16:45:05.785792 8225 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:46759 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.196.62:46759: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
I20260928 16:45:05.786728 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.777732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.30.196.62:46759)
0928 16:45:05.778073 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.778078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.778093 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:05.778134 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.778137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.778179 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.778279 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.779070 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.779589 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.780479 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.780708 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.780837 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.782338 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:05.782345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:05.782364 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:05.782401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:05.784238 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:05.784820 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:05.784825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:05.784830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:05.784951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:05.785246 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:05.785252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:05.785255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:05.785743 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:05.785957 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:05.786293 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 16:45:05.786562 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":82,"threads_started":1}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.791992 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.786306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:56663)
0928 16:45:05.786491 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.786503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.786566 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.786880 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.786885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.787057 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.787178 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.787187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.788118 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.788121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.788857 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.788868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.789461 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.789463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.789541 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.790058 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.790069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.791895 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 16:45:05.792148 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.786378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:56663 (local address 127.30.196.62:46759)
0928 16:45:05.786613 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.786617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.786629 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:45:05.786663 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.786666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.786706 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.786790 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.787319 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.788006 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.789023 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.789344 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.789453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.792034 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:56663: BlockingRecv error: recv got EOF from 127.30.196.3:56663 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:05 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:05.797411 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:05.792703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:48459)
0928 16:45:05.792817 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:05.792834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:05.792905 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:05.793127 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:05.793129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:05.793253 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:05.793324 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.793328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.793993 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.793995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.794500 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:05.794507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.794911 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.794914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:05.794970 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.795384 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:05.795394 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:05.797305 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:45:05.797576 7961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:05.792807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:48459 (local address 127.30.196.62:46759)
0928 16:45:05.792945 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:05.792949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:05.792960 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:45:05.792991 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:05.792993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:05.793027 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:05.793094 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:05.793423 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.793903 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.794610 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:05.794813 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:05.794892 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:05.797457 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:48459: BlockingRecv error: recv got EOF from 127.30.196.3:48459 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:45:05.797803 8225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:06.410567 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:06.404454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:49827)
0928 16:45:06.404737 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:06.404757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:06.404873 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:06.405167 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:06.405169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:06.405348 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:06.405449 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.405455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.406385 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.406388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.407242 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.407255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.407882 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.407885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.407949 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.408513 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:06.408524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:06.410386 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
W20260928 16:45:06.410696 8231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:06.404551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:49827 (local address 127.30.196.62:46759)
0928 16:45:06.404860 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:06.404864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:06.404878 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:06.404945 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:06.404947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:06.404990 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:06.405139 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:06.405543 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.406268 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.407355 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.407745 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.407845 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.410514 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:49827: BlockingRecv error: recv got EOF from 127.30.196.1:49827 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":141,"threads_started":1}
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:06.601611 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:06.596090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:37145)
0928 16:45:06.596404 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:06.596419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:06.596521 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:06.596788 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:06.596791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:06.596968 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:06.597069 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.597075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.597811 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.597813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.598371 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.598379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.598822 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.598824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.598885 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.599471 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:06.599483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:06.601440 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
W20260928 16:45:06.601720 8231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:06.596159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:37145 (local address 127.30.196.62:46759)
0928 16:45:06.596316 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:06.596322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:06.596513 (+ 191us) server_negotiation.cc:407] Connection header received
0928 16:45:06.596602 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:06.596605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:06.596655 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:06.596733 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:06.597178 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.597715 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.598482 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.598719 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.598799 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.601606 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:37145: BlockingRecv error: recv got EOF from 127.30.196.2:37145 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:06 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:06.803822 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:06.798404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:37421)
0928 16:45:06.798676 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:06.798690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:06.798786 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:06.799016 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:06.799019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:06.799205 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:06.799305 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.799310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.800051 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.800053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.800632 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:06.800640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.801087 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.801090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:06.801162 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.801687 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:06.801701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:06.803622 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260928 16:45:06.803891 8231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:06.798495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:37421 (local address 127.30.196.62:46759)
0928 16:45:06.798645 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:06.798648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:06.798740 (+ 92us) server_negotiation.cc:407] Connection header received
0928 16:45:06.798863 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:06.798865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:06.798909 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:06.798969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:06.799407 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.799944 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.800737 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:06.800946 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:06.801024 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:06.803795 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:37421: BlockingRecv error: recv got EOF from 127.30.196.3:37421 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:07.417588 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:07.411513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:50857)
0928 16:45:07.411891 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:07.411906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:07.412088 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:07.412360 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:07.412362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:07.412559 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:07.412661 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.412666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.413476 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.413478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.414044 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.414053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.414519 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.414522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.414593 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.415187 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:07.415202 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:07.417424 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
W20260928 16:45:07.417676 8235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:07.411594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:50857 (local address 127.30.196.62:46759)
0928 16:45:07.411905 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:07.411908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:07.412046 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:45:07.412189 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:07.412191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:07.412240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:07.412321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:07.412780 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.413357 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.414165 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.414386 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.414467 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.417564 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:50857: BlockingRecv error: recv got EOF from 127.30.196.1:50857 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:07.609484 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:07.602724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:50461)
0928 16:45:07.603031 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:07.603046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:07.603162 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:07.603433 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:07.603435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:07.603655 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:07.603797 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.603803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.604789 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.604792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.605558 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.605569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.606137 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.606140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.606208 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.606832 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:07.606856 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:07.609308 (+ 2452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
W20260928 16:45:07.609658 8235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:07.602857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:50461 (local address 127.30.196.62:46759)
0928 16:45:07.603031 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:07.603036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:07.603128 (+ 92us) server_negotiation.cc:407] Connection header received
0928 16:45:07.603255 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:07.603257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:07.603306 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:07.603389 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:07.603952 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.604644 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.605725 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.605998 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.606106 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.609490 (+ 3384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:50461: BlockingRecv error: recv got EOF from 127.30.196.2:50461 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:07 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:07.818806 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:07.804732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:56033)
0928 16:45:07.805182 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:07.805206 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:07.805347 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:07.805655 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:07.805657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:07.805847 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:07.805949 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.805956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.806798 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.806801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.807372 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:07.807381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.807851 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.807854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:07.807921 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.816737 (+ 8816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:07.816752 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:07.818648 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
W20260928 16:45:07.818938 8235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:07.804926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:56033 (local address 127.30.196.62:46759)
0928 16:45:07.805091 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:07.805095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:07.805313 (+ 218us) server_negotiation.cc:407] Connection header received
0928 16:45:07.805480 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:07.805482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:07.805529 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:07.805625 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:07.806065 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.806663 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.807487 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:07.807746 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:07.807829 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:07.818808 (+ 10979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:56033: BlockingRecv error: recv got EOF from 127.30.196.3:56033 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:08.425316 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:08.418652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:55119)
0928 16:45:08.418952 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:08.418966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:08.419068 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:08.419406 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:08.419409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:08.419603 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:08.419790 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.419798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.420808 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.420811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.421759 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.421772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.422296 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.422300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.422380 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.422995 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:08.423008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:08.425183 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
W20260928 16:45:08.425530 8239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:08.418770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:55119 (local address 127.30.196.62:46759)
0928 16:45:08.419136 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:08.419139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:08.419154 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:08.419208 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:08.419211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:08.419254 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:08.419345 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:08.419931 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.420706 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.421876 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.422167 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.422253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.425347 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:55119: BlockingRecv error: recv got EOF from 127.30.196.1:55119 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:08.616290 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:08.610351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:52895)
0928 16:45:08.610741 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:08.610755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:08.610873 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:08.611188 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:08.611190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:08.611375 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:08.611487 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.611492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.612374 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.612377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.612953 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.612961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.613455 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.613458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.613547 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.614111 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:08.614123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:08.616110 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
W20260928 16:45:08.616353 8239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:08.610480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:52895 (local address 127.30.196.62:46759)
0928 16:45:08.610733 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:08.610737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:08.610854 (+ 117us) server_negotiation.cc:407] Connection header received
0928 16:45:08.610964 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:08.610966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:08.611061 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:08.611145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:08.611601 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.612252 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.613098 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.613335 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.613455 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.616243 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:52895: BlockingRecv error: recv got EOF from 127.30.196.2:52895 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:08 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:08.825599 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:08.819799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:42113)
0928 16:45:08.820125 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:08.820139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:08.820242 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:08.820518 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:08.820520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:08.820705 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:08.820802 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.820807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.821651 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.821655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.822278 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:08.822287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.822816 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.822820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:08.822891 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.823503 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:08.823515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:08.825473 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":126,"threads_started":1}
W20260928 16:45:08.825812 8239 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:08.819819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:42113 (local address 127.30.196.62:46759)
0928 16:45:08.819987 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:08.819992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:08.820218 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:45:08.820320 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:08.820323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:08.820385 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:08.820482 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:08.820907 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.821510 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.822416 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:08.822676 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:08.822767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:08.825658 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:42113: BlockingRecv error: recv got EOF from 127.30.196.3:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:09.432361 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:09.426230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:51027)
0928 16:45:09.426554 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:09.426569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:09.426678 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:09.427006 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:09.427008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:09.427219 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:09.427328 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.427334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.428390 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.428393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.429126 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.429141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.429621 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.429624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.429689 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.430174 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:09.430186 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:09.432226 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
W20260928 16:45:09.432619 8243 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:09.426344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:51027 (local address 127.30.196.62:46759)
0928 16:45:09.426590 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:09.426593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:09.426625 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:45:09.426768 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:09.426771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:09.426816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:09.426906 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:09.427480 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.428248 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.429250 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.429512 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.429595 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.432374 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:51027: BlockingRecv error: recv got EOF from 127.30.196.1:51027 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:09.623154 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:09.617115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:38003)
0928 16:45:09.617571 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:09.617588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:09.617710 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:09.618008 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:09.618011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:09.618255 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:09.618352 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.618359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.619252 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.619256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.620047 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.620056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.620575 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.620578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.620644 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.621206 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:09.621220 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:09.623027 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":132,"threads_started":1}
W20260928 16:45:09.623296 8243 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:09.617307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:38003 (local address 127.30.196.62:46759)
0928 16:45:09.617577 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:09.617584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:09.617657 (+ 73us) server_negotiation.cc:407] Connection header received
0928 16:45:09.617834 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:09.617837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:09.617895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:09.617985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:09.618482 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.619140 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.620193 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.620483 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.620567 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.623177 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:38003: BlockingRecv error: recv got EOF from 127.30.196.2:38003 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:09 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:09.832298 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:09.826469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:36173)
0928 16:45:09.826926 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:09.826947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:09.827090 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:09.827365 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:09.827368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:09.827586 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:09.827713 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.827719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.828528 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.828534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.829134 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:09.829143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.829599 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.829603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:09.829689 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.830211 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:09.830224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:09.832177 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":101,"threads_started":1}
W20260928 16:45:09.832490 8243 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:09.826623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:36173 (local address 127.30.196.62:46759)
0928 16:45:09.826888 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:09.826893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:09.827026 (+ 133us) server_negotiation.cc:407] Connection header received
0928 16:45:09.827194 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:09.827197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:09.827252 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:09.827342 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:09.827837 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.828410 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.829261 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:09.829481 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:09.829593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:09.832319 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:36173: BlockingRecv error: recv got EOF from 127.30.196.3:36173 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:10.439422 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:10.433224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:59885)
0928 16:45:10.433524 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:10.433537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:10.433628 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:10.433887 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:10.433889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:10.434087 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:10.434257 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.434264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.435302 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.435306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.436240 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.436253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.436731 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.436733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.436795 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.437343 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:10.437355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:10.439305 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
W20260928 16:45:10.439621 8247 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:10.433307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:59885 (local address 127.30.196.62:46759)
0928 16:45:10.433587 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:10.433591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:10.433606 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:45:10.433719 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:10.433721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:10.433765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:10.433848 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:10.434442 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.435145 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.436364 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.436626 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.436708 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.439446 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:59885: BlockingRecv error: recv got EOF from 127.30.196.1:59885 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:10.630267 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:10.624262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:47863)
0928 16:45:10.624563 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:10.624579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:10.624676 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:10.624947 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:10.624949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:10.625157 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:10.625252 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.625258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.626106 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.626108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.626806 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.626816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.627420 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.627423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.627516 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.628118 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:10.628131 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:10.630074 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
W20260928 16:45:10.630321 8247 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:10.624497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:47863 (local address 127.30.196.62:46759)
0928 16:45:10.624654 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:10.624658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:10.624675 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:45:10.624773 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:10.624776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:10.624821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:10.624918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:10.625368 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.625986 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.626939 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.627267 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.627405 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.630213 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:47863: BlockingRecv error: recv got EOF from 127.30.196.2:47863 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:10 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:10.840879 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:10.833085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:44189)
0928 16:45:10.833353 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:10.833375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:10.833506 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:10.833847 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:10.833850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:10.834113 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:10.834251 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.834258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.835546 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.835553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.836627 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:10.836639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.837356 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.837360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:10.837466 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.838177 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:10.838196 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:10.840684 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
W20260928 16:45:10.841267 8247 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:10.833173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:44189 (local address 127.30.196.62:46759)
0928 16:45:10.833329 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:10.833335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:10.833464 (+ 129us) server_negotiation.cc:407] Connection header received
0928 16:45:10.833618 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:10.833623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:10.833695 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:10.833799 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:10.834402 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.835403 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.836786 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:10.837207 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:10.837338 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:10.840947 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:44189: BlockingRecv error: recv got EOF from 127.30.196.3:44189 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:11.446046 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:11.440298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:59293)
0928 16:45:11.440587 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:11.440600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:11.440687 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:11.440915 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:11.440917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:11.441152 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:11.441257 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.441263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.442013 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.442015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.442607 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.442617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.443055 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.443057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.443126 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.443878 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:11.443890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:11.445861 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
W20260928 16:45:11.446153 8251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:11.440381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:59293 (local address 127.30.196.62:46759)
0928 16:45:11.440626 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:11.440631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:11.440645 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:11.440765 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:11.440767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:11.440812 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:11.440880 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:11.441366 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.441898 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.442716 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.442934 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.443012 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.446012 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:59293: BlockingRecv error: recv got EOF from 127.30.196.1:59293 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:11.637529 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:11.631219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:47115)
0928 16:45:11.631521 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:11.631535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:11.631715 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:11.632054 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:11.632058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:11.632304 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:11.632419 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.632425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.633264 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.633269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.633933 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.633942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.634482 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.634486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.634586 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.635233 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:11.635247 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:11.637331 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260928 16:45:11.637609 8251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:11.631255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:47115 (local address 127.30.196.62:46759)
0928 16:45:11.631402 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:11.631406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:11.631638 (+ 232us) server_negotiation.cc:407] Connection header received
0928 16:45:11.631852 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:11.631861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:11.631914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:11.632009 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:11.632553 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.633139 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.634094 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.634349 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.634478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.637486 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:47115: BlockingRecv error: recv got EOF from 127.30.196.2:47115 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:11 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:11.848273 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:11.841834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:53873)
0928 16:45:11.842196 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:11.842212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:11.842374 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:11.842729 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:11.842732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:11.842962 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:11.843084 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.843092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.844105 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.844111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.844836 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:11.844847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.845336 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.845339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:11.845412 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.845926 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:11.845943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:11.848148 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
W20260928 16:45:11.848439 8251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:11.841964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:53873 (local address 127.30.196.62:46759)
0928 16:45:11.842167 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:11.842172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:11.842312 (+ 140us) server_negotiation.cc:407] Connection header received
0928 16:45:11.842544 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:11.842547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:11.842609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:11.842702 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:11.843224 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.843990 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.844950 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:11.845193 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:11.845281 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:11.848309 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:53873: BlockingRecv error: recv got EOF from 127.30.196.3:53873 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:12.452349 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:12.446908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:49487)
0928 16:45:12.447225 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:12.447239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:12.447328 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:12.447567 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:12.447569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:12.447793 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:12.447892 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.447898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.448640 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.448643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.449217 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.449225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.449664 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.449666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.449728 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.450323 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:12.450336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:12.452231 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
W20260928 16:45:12.452641 8255 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:12.447009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:49487 (local address 127.30.196.62:46759)
0928 16:45:12.447278 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:12.447282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:12.447295 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:45:12.447407 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:12.447409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:12.447452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:12.447528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:12.447992 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.448530 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.449324 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.449543 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.449645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.452484 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:49487: BlockingRecv error: recv got EOF from 127.30.196.1:49487 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:12.644368 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:12.638853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:46893)
0928 16:45:12.639198 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:12.639211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:12.639315 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:12.639735 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:12.639737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:12.639923 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:12.640021 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.640027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.640768 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.640770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.641331 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.641339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.641758 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.641761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.641820 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.642305 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:12.642316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:12.644247 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
W20260928 16:45:12.644551 8255 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:12.638897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:46893 (local address 127.30.196.62:46759)
0928 16:45:12.639035 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:12.639038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:12.639281 (+ 243us) server_negotiation.cc:407] Connection header received
0928 16:45:12.639472 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:12.639478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:12.639541 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:12.639617 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:12.640139 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.640669 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.641442 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.641657 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.641737 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.644416 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:46893: BlockingRecv error: recv got EOF from 127.30.196.2:46893 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:12 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:12.861052 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:12.849218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:55273)
0928 16:45:12.849610 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:12.849623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:12.849758 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:12.850054 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:12.850057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:12.850288 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:12.850379 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.850385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.851152 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.851157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.851776 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:12.851785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.852501 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.852506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:12.852615 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.858811 (+ 6196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:12.858828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:12.860877 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":88,"threads_started":1}
W20260928 16:45:12.861137 8255 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:12.849337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55273 (local address 127.30.196.62:46759)
0928 16:45:12.849515 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:12.849520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:12.849703 (+ 183us) server_negotiation.cc:407] Connection header received
0928 16:45:12.849883 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:12.849886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:12.849942 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:12.850026 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:12.850501 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.851035 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.851899 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:12.852141 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:12.852502 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:12.861023 (+ 8521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:55273: BlockingRecv error: recv got EOF from 127.30.196.3:55273 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:13.458338 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:13.453135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:42643)
0928 16:45:13.453418 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:13.453431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:13.453518 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:13.453748 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:13.453750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:13.453943 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:13.454043 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.454049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.454782 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.454784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.455348 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.455356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.455835 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.455838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.455938 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.456447 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:13.456458 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:13.458221 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
W20260928 16:45:13.458498 8259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:13.453298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:42643 (local address 127.30.196.62:46759)
0928 16:45:13.453537 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:13.453540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:13.453552 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:45:13.453589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:13.453591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:13.453633 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:13.453718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:13.454164 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.454690 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.455447 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.455724 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.455815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.458362 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:42643: BlockingRecv error: recv got EOF from 127.30.196.1:42643 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:13.651350 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:13.645465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:41447)
0928 16:45:13.645724 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:13.645739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:13.645852 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:13.646210 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:13.646212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:13.646389 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:13.646486 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.646491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.647459 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.647462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.648125 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.648135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.648610 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.648614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.648711 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.649390 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:13.649403 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:13.651233 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":113,"threads_started":1}
W20260928 16:45:13.651495 8259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:13.645521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:41447 (local address 127.30.196.62:46759)
0928 16:45:13.645698 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:13.645702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:13.645827 (+ 125us) server_negotiation.cc:407] Connection header received
0928 16:45:13.646036 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:13.646039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:13.646090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:13.646170 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:13.646600 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.647358 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.648241 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.648495 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.648576 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.651364 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:41447: BlockingRecv error: recv got EOF from 127.30.196.2:41447 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:13 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:13.867295 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:13.861937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:53339)
0928 16:45:13.862207 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:13.862223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:13.862340 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:13.862588 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:13.862600 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:13.862779 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:13.862885 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.862890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.863644 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.863646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.864235 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:13.864243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.864732 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.864734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:13.864795 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.865304 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:13.865315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:13.867178 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260928 16:45:13.867529 8259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:13.861970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:53339 (local address 127.30.196.62:46759)
0928 16:45:13.862116 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:13.862124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:13.862308 (+ 184us) server_negotiation.cc:407] Connection header received
0928 16:45:13.862415 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:13.862417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:13.862485 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:13.862553 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:13.863012 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.863540 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.864382 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:13.864624 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:13.864717 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:13.867379 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:53339: BlockingRecv error: recv got EOF from 127.30.196.3:53339 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:14.464815 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:14.459267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:49853)
0928 16:45:14.459591 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:14.459605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:14.459772 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:14.460054 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:14.460057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:14.460240 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:14.460338 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.460344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.461104 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.461106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.461674 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.461682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.462125 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.462128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.462188 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.462709 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:14.462720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:14.464686 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
W20260928 16:45:14.465077 8263 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:14.459399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:49853 (local address 127.30.196.62:46759)
0928 16:45:14.459651 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:14.459655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:14.459688 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:45:14.459869 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:14.459871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:14.459915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:14.459992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:14.460446 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.460988 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.461780 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.462005 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.462080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.464864 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:49853: BlockingRecv error: recv got EOF from 127.30.196.1:49853 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:14.657814 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:14.652376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:35213)
0928 16:45:14.652698 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:14.652712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:14.652818 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:14.653065 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:14.653067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:14.653258 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:14.653360 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.653365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.654104 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.654107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.654687 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.654695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.655121 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.655124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.655184 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.655798 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:14.655811 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:14.657672 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
W20260928 16:45:14.657984 8263 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:14.652445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:35213 (local address 127.30.196.62:46759)
0928 16:45:14.652601 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:14.652606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:14.652789 (+ 183us) server_negotiation.cc:407] Connection header received
0928 16:45:14.652902 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:14.652904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:14.652952 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:14.653022 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:14.653465 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.653996 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.654803 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.655020 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.655117 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.657824 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:35213: BlockingRecv error: recv got EOF from 127.30.196.2:35213 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:14 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:14.874578 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:14.868218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:39331)
0928 16:45:14.868610 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:14.868625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:14.868784 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:14.869189 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:14.869192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:14.869433 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:14.869533 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.869539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.870340 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.870346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.870922 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:14.870931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.871434 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.871438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:14.871528 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.872174 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:14.872188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:14.874413 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
W20260928 16:45:14.874835 8263 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:14.868360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:39331 (local address 127.30.196.62:46759)
0928 16:45:14.868571 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:14.868576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:14.868720 (+ 144us) server_negotiation.cc:407] Connection header received
0928 16:45:14.868997 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:14.869000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:14.869052 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:14.869158 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:14.869666 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.870221 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.871050 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:14.871302 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:14.871419 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:14.874686 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:39331: BlockingRecv error: recv got EOF from 127.30.196.3:39331 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:15.471400 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:15.465659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:56735)
0928 16:45:15.465949 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:15.465962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:15.466059 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:15.466357 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:15.466359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:15.466541 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:15.466663 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.466669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.467415 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.467417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.468032 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.468042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.468679 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.468684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.468775 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.469356 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:15.469373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:15.471281 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
W20260928 16:45:15.471558 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:15.465815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:56735 (local address 127.30.196.62:46759)
0928 16:45:15.466043 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:15.466047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:15.466061 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:15.466189 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:15.466192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:15.466237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:15.466315 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:15.466765 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.467322 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.468183 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.468539 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.468647 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.471419 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:56735: BlockingRecv error: recv got EOF from 127.30.196.1:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:15.664719 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:15.658866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:35369)
0928 16:45:15.659215 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:15.659229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:15.659350 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:15.659631 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:15.659633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:15.659883 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:15.659988 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.659994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.660768 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.660771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.661385 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.661393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.661860 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.661862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.661950 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.662528 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:15.662541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:15.664520 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":134,"threads_started":1}
W20260928 16:45:15.664757 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:15.658936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:35369 (local address 127.30.196.62:46759)
0928 16:45:15.659147 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:15.659154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:15.659307 (+ 153us) server_negotiation.cc:407] Connection header received
0928 16:45:15.659466 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:15.659470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:15.659529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:15.659596 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:15.660089 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.660641 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.661502 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.661728 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.661812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.664654 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:35369: BlockingRecv error: recv got EOF from 127.30.196.2:35369 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:15 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:15.881078 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:15.875384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:42983)
0928 16:45:15.875763 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:15.875781 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:15.875934 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:15.876229 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:15.876232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:15.876447 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:15.876549 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.876555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.877374 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.877379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.877942 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:15.877950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.878432 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.878435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:15.878523 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.879087 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:15.879100 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:15.880956 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
W20260928 16:45:15.881232 8267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:15.875564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:42983 (local address 127.30.196.62:46759)
0928 16:45:15.875762 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:15.875768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:15.875869 (+ 101us) server_negotiation.cc:407] Connection header received
0928 16:45:15.876037 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:15.876040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:15.876102 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:15.876202 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:15.876696 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.877255 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.878078 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:15.878309 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:15.878424 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:15.881107 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:42983: BlockingRecv error: recv got EOF from 127.30.196.3:42983 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:16.478046 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:16.472325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:57351)
0928 16:45:16.472605 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:16.472624 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:16.472727 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:16.472984 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:16.472986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:16.473221 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:16.473338 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.473345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.474071 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.474074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.474630 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.474638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.475074 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.475078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.475157 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.475642 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:16.475654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:16.477931 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
W20260928 16:45:16.478214 8272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:16.472388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:57351 (local address 127.30.196.62:46759)
0928 16:45:16.472663 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:16.472667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:16.472680 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:45:16.472819 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:16.472821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:16.472865 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:16.472947 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:16.473444 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.473982 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.474729 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.474942 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.475019 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.478076 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:57351: BlockingRecv error: recv got EOF from 127.30.196.1:57351 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:16.671034 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:16.665558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:44155)
0928 16:45:16.665875 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:16.665889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:16.666008 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:16.666252 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:16.666253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:16.666437 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:16.666534 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.666539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.667284 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.667287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.667930 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.667940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.668380 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.668382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.668443 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.668967 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:16.668979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:16.670908 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
W20260928 16:45:16.671237 8272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:16.665644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:44155 (local address 127.30.196.62:46759)
0928 16:45:16.665811 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:16.665818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:16.665981 (+ 163us) server_negotiation.cc:407] Connection header received
0928 16:45:16.666097 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:16.666099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:16.666145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:16.666211 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:16.666637 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.667184 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.668038 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.668282 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.668360 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.671068 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:44155: BlockingRecv error: recv got EOF from 127.30.196.2:44155 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:16 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:45:16.887946 8272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:16.882046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:41925 (local address 127.30.196.62:46759)
0928 16:45:16.882207 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:16.882212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:16.882417 (+ 205us) server_negotiation.cc:407] Connection header received
0928 16:45:16.882530 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:16.882533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:16.882580 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:16.882657 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:16.883131 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.883698 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.884556 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.884872 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.884954 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.887847 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:41925: BlockingRecv error: recv got EOF from 127.30.196.3:41925 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:45:16.887953 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:16.881962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:41925)
0928 16:45:16.882309 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:16.882323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:16.882433 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:16.882728 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:16.882731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:16.882914 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:16.883014 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.883018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.883871 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.883874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.884437 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:16.884445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:16.884979 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:16.884981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:16.885043 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:16.885558 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:16.885569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:16.887712 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":146,"threads_started":1}
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:17.485047 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:17.478912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:35005)
0928 16:45:17.479228 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:17.479241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:17.479335 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:17.479645 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:17.479647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:17.479858 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:17.480033 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.480040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.480956 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.480959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.481515 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.481525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.482036 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.482038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.482102 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.482611 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:17.482623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:17.484867 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
W20260928 16:45:17.485124 8276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:17.479063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:35005 (local address 127.30.196.62:46759)
0928 16:45:17.479306 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:17.479310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:17.479324 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:17.479405 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:17.479407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:17.479470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:17.479563 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:17.480231 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.480794 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.481647 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.481882 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.481969 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.484996 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:35005: BlockingRecv error: recv got EOF from 127.30.196.1:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:17.677778 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:17.671982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:55911)
0928 16:45:17.672288 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:17.672302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:17.672404 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:17.672676 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:17.672678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:17.672869 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:17.673070 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.673080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.673909 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.673912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.674503 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.674514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.675017 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.675020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.675110 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.675741 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:17.675760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:17.677661 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
W20260928 16:45:17.677970 8276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:17.672103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:55911 (local address 127.30.196.62:46759)
0928 16:45:17.672273 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:17.672276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:17.672376 (+ 100us) server_negotiation.cc:407] Connection header received
0928 16:45:17.672488 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:17.672506 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:17.672553 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:17.672633 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:17.673229 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.673787 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.674649 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.674872 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.674974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.677824 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:55911: BlockingRecv error: recv got EOF from 127.30.196.2:55911 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:17 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:17.899658 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:17.888861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:60451)
0928 16:45:17.889188 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:17.889202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:17.889309 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:17.889542 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:17.889544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:17.889727 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:17.889822 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.889828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.890660 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.890666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.891232 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:17.891240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.891860 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.891864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:17.891951 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.897415 (+ 5464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:17.897430 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:17.899452 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
W20260928 16:45:17.899736 8276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:17.888942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:60451 (local address 127.30.196.62:46759)
0928 16:45:17.889120 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:17.889125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:17.889259 (+ 134us) server_negotiation.cc:407] Connection header received
0928 16:45:17.889389 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:17.889392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:17.889439 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:17.889499 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:17.889928 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.890536 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.891357 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:17.891728 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:17.891854 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:17.899603 (+ 7749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:60451: BlockingRecv error: recv got EOF from 127.30.196.3:60451 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:18.494330 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:18.485948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:46743)
0928 16:45:18.486336 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:18.486355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:18.486516 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:18.486941 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:18.486945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:18.487221 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:18.487458 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.487478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.488659 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.488664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.489945 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.489967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.490668 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.490673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.490821 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.491530 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:18.491604 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:18.494106 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
W20260928 16:45:18.494474 8280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:18.486019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:46743 (local address 127.30.196.62:46759)
0928 16:45:18.486361 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:18.486365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:18.486439 (+ 74us) server_negotiation.cc:407] Connection header received
0928 16:45:18.486706 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:18.486708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:18.486778 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:18.486907 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:18.487641 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.488467 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.490186 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.490513 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.490612 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.494295 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:46743: BlockingRecv error: recv got EOF from 127.30.196.1:46743 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:18.688779 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:18.678876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:33441)
0928 16:45:18.679269 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:18.679307 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:18.679432 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:18.679831 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:18.679837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:18.680175 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:18.680352 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.680359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.681532 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.681535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.682539 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.682558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.683309 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.683313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.683440 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.684402 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:18.684425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:18.688414 (+ 3989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
W20260928 16:45:18.689065 8280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:18.679089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33441 (local address 127.30.196.62:46759)
0928 16:45:18.679325 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:18.679328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:18.679389 (+ 61us) server_negotiation.cc:407] Connection header received
0928 16:45:18.679541 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:18.679543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:18.679621 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:18.679775 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:18.680515 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.681343 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.682735 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.683094 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.683229 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.688722 (+ 5493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:33441: BlockingRecv error: recv got EOF from 127.30.196.2:33441 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:18 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:18.906005 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:18.900561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:58833)
0928 16:45:18.900838 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:18.900858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:18.900966 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:18.901213 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:18.901215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:18.901400 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:18.901495 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.901500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.902215 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.902218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.902770 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:18.902778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.903170 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.903172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:18.903231 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.903825 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:18.903837 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:18.905833 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
W20260928 16:45:18.906059 8280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:18.900707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:58833 (local address 127.30.196.62:46759)
0928 16:45:18.900867 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:18.900870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:18.900935 (+ 65us) server_negotiation.cc:407] Connection header received
0928 16:45:18.901049 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:18.901052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:18.901099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:18.901171 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:18.901610 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.902120 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.902870 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:18.903074 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:18.903147 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:18.905952 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:58833: BlockingRecv error: recv got EOF from 127.30.196.3:58833 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:19.501428 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:19.495391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:39403)
0928 16:45:19.495748 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:19.495764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:19.495868 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:19.496166 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:19.496169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:19.496369 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:19.496545 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.496554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.497586 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.497590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.498288 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.498299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.498797 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.498801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.498892 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.499370 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:19.499381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:19.501246 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
W20260928 16:45:19.501502 8284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:19.495464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:39403 (local address 127.30.196.62:46759)
0928 16:45:19.495807 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:19.495811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:19.495825 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:45:19.495967 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:19.495969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:19.496015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:19.496129 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:19.496707 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.497473 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.498410 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.498663 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.498778 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.501370 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:39403: BlockingRecv error: recv got EOF from 127.30.196.1:39403 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":64,"threads_started":1}
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:19.696529 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:19.690235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:51009)
0928 16:45:19.690557 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:19.690571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:19.690679 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:19.690945 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:19.690947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:19.691192 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:19.691297 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.691304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.692325 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.692329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.693146 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.693157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.693593 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.693596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.693662 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.694264 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:19.694286 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:19.696382 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
W20260928 16:45:19.696677 8284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:19.690319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:51009 (local address 127.30.196.62:46759)
0928 16:45:19.690468 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:19.690472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:19.690645 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:45:19.690768 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:19.690771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:19.690817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:19.690904 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:19.691433 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.692183 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.693261 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.693482 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.693561 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.696525 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:51009: BlockingRecv error: recv got EOF from 127.30.196.2:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:19 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.3@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:19.913136 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:19.907097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.3:41137)
0928 16:45:19.907373 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:19.907399 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:19.907503 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:19.907821 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:19.907823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:19.908045 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:19.908148 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.908154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.908971 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.908974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.909556 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:19.909566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.910087 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.910090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:19.910156 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.910644 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:19.910663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:19.913003 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260928 16:45:19.913275 8284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:19.907166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:41137 (local address 127.30.196.62:46759)
0928 16:45:19.907339 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:19.907343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:19.907475 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:45:19.907590 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:19.907593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:19.907657 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:19.907771 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:19.908284 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.908829 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.909699 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:19.909927 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:19.910066 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:19.913150 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.3:41137: BlockingRecv error: recv got EOF from 127.30.196.3:41137 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 16:45:20 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:20 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.1@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:20.508467 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:20.502574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.1:32793)
0928 16:45:20.502851 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:20.502865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:20.502957 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:20.503221 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:20.503223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:20.503404 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:20.503527 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:20.503533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.504399 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.504401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:20.505241 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:20.505254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.505881 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.505884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:20.505952 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:20.506545 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:20.506557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:20.508333 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
W20260928 16:45:20.508608 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:20.502654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:32793 (local address 127.30.196.62:46759)
0928 16:45:20.502900 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:20.502904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:20.502920 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:45:20.503047 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:20.503051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:20.503098 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:20.503186 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:20.503633 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.504295 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.505394 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.505770 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.505857 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:20.508477 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.1:32793: BlockingRecv error: recv got EOF from 127.30.196.1:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
Sep 28 16:45:20 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:45:20 dist-test-slave-5rg1 krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.196.2@KRBTEST.COM for oryx/127.30.196.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:45:20.709337 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:20.697683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46759 (local address 127.30.196.2:45067)
0928 16:45:20.698103 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:20.698116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:20.698245 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:20.698565 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:20.698570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:20.698849 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:20.698957 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:20.698964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.699794 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.699797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:20.700391 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:20.700400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.700844 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.700846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:20.700909 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:20.706994 (+ 6085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:20.707009 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:20.709159 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.196.62:46759: Server oryx/127.30.196.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":84,"threads_started":1}
W20260928 16:45:20.709609 8288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:20.697756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:45067 (local address 127.30.196.62:46759)
0928 16:45:20.698076 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:20.698083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:20.698221 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:45:20.698348 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:20.698351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:20.698402 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:20.698508 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:20.699130 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.699695 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.700512 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:20.700746 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:20.700830 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:20.709315 (+ 8485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.196.2:45067: BlockingRecv error: recv got EOF from 127.30.196.2:45067 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:20.780550 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7809
I20260928 16:45:20.786700 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7962
I20260928 16:45:20.792976 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8094
I20260928 16:45:20.798285 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 7738
2026-09-28T16:45:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18057 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T16:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:20Z Disabled control of system clock
I20260928 16:45:20.830856 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:39727
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:45:20.965287 8293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:20.965587 8293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:20.965623 8293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:20.969892 8293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:20.969969 8293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:20.969985 8293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:20.970003 8293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:20.970058 8293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:20.975032 8293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:39727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:20.976264 8293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:20.977710 8293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:45:20.987298 8293 server_base.cc:1034] running on GCE node
W20260928 16:45:20.987246 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:20.987236 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:20.987239 8301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:20.988183 8293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:20.989367 8293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:20.990562 8293 hybrid_clock.cc:648] HybridClock initialized: now 1790613920990541 us; error 34 us; skew 500 ppm
I20260928 16:45:20.992532 8293 webserver.cc:460] Webserver started at http://127.30.196.62:38203/ using document root <none> and password file <none>
I20260928 16:45:20.993160 8293 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:20.993227 8293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:20.993474 8293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:20.995580 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "5a5b11924bf14457a44453b93570c197"
format_stamp: "Formatted at 2026-09-28 16:45:20 on dist-test-slave-5rg1"
server_key: "94e9d379332417e59ad5ad84aeea6022"
server_key_iv: "0b570567ef3e40babb3a68524aeac97e"
server_key_version: "encryptionkey@0"
I20260928 16:45:20.996227 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "5a5b11924bf14457a44453b93570c197"
format_stamp: "Formatted at 2026-09-28 16:45:20 on dist-test-slave-5rg1"
server_key: "94e9d379332417e59ad5ad84aeea6022"
server_key_iv: "0b570567ef3e40babb3a68524aeac97e"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.000183 8293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:45:21.002606 8307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.003921 8293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:21.004060 8293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "5a5b11924bf14457a44453b93570c197"
format_stamp: "Formatted at 2026-09-28 16:45:20 on dist-test-slave-5rg1"
server_key: "94e9d379332417e59ad5ad84aeea6022"
server_key_iv: "0b570567ef3e40babb3a68524aeac97e"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.004179 8293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:21.015908 8293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:21.016614 8293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:21.016817 8293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:21.024281 8293 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:39727
I20260928 16:45:21.024353 8359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:39727 every 8 connection(s)
I20260928 16:45:21.025327 8293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:21.027549 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8293
I20260928 16:45:21.027640 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:45:21.027956 31504 external_mini_cluster.cc:1428] Setting key bec3f953190e3dcfb0ff87ae84c04a08
I20260928 16:45:21.028924 8360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:21.035792 8360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197: Bootstrap starting.
I20260928 16:45:21.037554 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.029464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.30.196.62:39727)
0928 16:45:21.030104 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.030122 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.030157 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:45:21.030826 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.030841 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.031232 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.031511 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.031926 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.032924 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.034110 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.034586 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.034715 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.035641 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:21.035647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:21.035714 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:21.036188 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:21.036247 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:21.036270 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:21.036348 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:21.036510 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:45:21.036832 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260928 16:45:21.038231 8360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:21.038961 8360 log.cc:824] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:21.040864 8360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197: No bootstrap required, opened a new log
I20260928 16:45:21.044420 8360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } }
I20260928 16:45:21.044672 8360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:21.044749 8360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5b11924bf14457a44453b93570c197, State: Initialized, Role: FOLLOWER
I20260928 16:45:21.045188 8360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } }
I20260928 16:45:21.045356 8360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:21.045444 8360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:21.045557 8360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:21.046542 8360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } }
I20260928 16:45:21.046849 8360 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5b11924bf14457a44453b93570c197; no voters:
I20260928 16:45:21.047127 8360 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:45:21.047250 8365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:21.047521 8365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5b11924bf14457a44453b93570c197, State: Running, Role: LEADER
I20260928 16:45:21.047896 8365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } }
I20260928 16:45:21.048432 8360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:21.049612 8366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5b11924bf14457a44453b93570c197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } } }
I20260928 16:45:21.049669 8367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5b11924bf14457a44453b93570c197. Latest consensus state: current_term: 1 leader_uuid: "5a5b11924bf14457a44453b93570c197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b11924bf14457a44453b93570c197" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39727 } } }
I20260928 16:45:21.049737 8366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:21.049737 8367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:21.050244 8374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:21.053071 8374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:21.058848 8374 catalog_manager.cc:1347] Generated new cluster ID: 489520d23f034606a5f89cf59eed620f
I20260928 16:45:21.058940 8374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:21.071357 8374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:45:21.072629 8374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:21.078017 8374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5b11924bf14457a44453b93570c197: Generated new TSK 0
I20260928 16:45:21.078861 8374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:45:21.086694 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--builtin_ntp_servers=127.30.196.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 16:45:21.203146 8384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:21.203426 8384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:21.203465 8384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:21.207829 8384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:21.207896 8384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:21.207978 8384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:45:21.213063 8384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:21.214228 8384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:21.215634 8384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:21.225518 8390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.225678 8384 server_base.cc:1034] running on GCE node
W20260928 16:45:21.225530 8391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.225555 8393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.226455 8384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:21.227236 8384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:21.228462 8384 hybrid_clock.cc:648] HybridClock initialized: now 1790613921228431 us; error 50 us; skew 500 ppm
I20260928 16:45:21.230408 8384 webserver.cc:460] Webserver started at http://127.30.196.1:39021/ using document root <none> and password file <none>
I20260928 16:45:21.231002 8384 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:21.231093 8384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:21.231323 8384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:21.233471 8384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "0f4fb6229a6147cd8c35f262288c21dc"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "6bdb90427f4699d339e9bf6d3458cf9c"
server_key_iv: "a5bda31e3e3f9ca6c56fe9b752e42652"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.234048 8384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "0f4fb6229a6147cd8c35f262288c21dc"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "6bdb90427f4699d339e9bf6d3458cf9c"
server_key_iv: "a5bda31e3e3f9ca6c56fe9b752e42652"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.237759 8384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:45:21.240324 8399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.241518 8384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:21.241662 8384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "0f4fb6229a6147cd8c35f262288c21dc"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "6bdb90427f4699d339e9bf6d3458cf9c"
server_key_iv: "a5bda31e3e3f9ca6c56fe9b752e42652"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.241780 8384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:21.267908 8384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:21.269097 8384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:21.269321 8384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:21.269929 8384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:21.270910 8384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:21.270959 8384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.271003 8384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:21.271023 8384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.279857 8384 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:35909
I20260928 16:45:21.279881 8512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:35909 every 8 connection(s)
I20260928 16:45:21.280828 8384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:21.283783 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8384
I20260928 16:45:21.283905 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:45:21.284283 31504 external_mini_cluster.cc:1428] Setting key 41f1ba68556cb3f913c395471e72e5b6
I20260928 16:45:21.286691 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--builtin_ntp_servers=127.30.196.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:45:21.291518 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.282809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:55755 (local address 127.30.196.62:39727)
0928 16:45:21.282972 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.282977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.284369 (+ 1392us) server_negotiation.cc:407] Connection header received
0928 16:45:21.285995 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.285998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.286064 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.286146 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.287403 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.288056 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.289346 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.289562 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.289656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.291012 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:21.291015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:21.291017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:21.291031 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:21.291053 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:21.291061 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:21.291127 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:21.291320 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 16:45:21.291396 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:45:21.293030 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.283250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39727 (local address 127.30.196.1:55755)
0928 16:45:21.284127 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:21.284190 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:21.285737 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:21.286353 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:21.286363 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:21.286479 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:21.287238 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.287262 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.288213 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.288216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.289216 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.289231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.289728 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.289735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.289977 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.290671 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:21.290675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:21.290738 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:21.290839 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:21.291151 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:21.291164 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:21.291172 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:45:21.291323 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 16:45:21.291735 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260928 16:45:21.294471 8513 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39727
I20260928 16:45:21.294729 8513 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:21.295277 8513 heartbeater.cc:511] Master 127.30.196.62:39727 requested a full tablet report, sending...
I20260928 16:45:21.297300 8324 ts_manager.cc:194] Registered new tserver with Master: 0f4fb6229a6147cd8c35f262288c21dc (127.30.196.1:35909)
I20260928 16:45:21.299057 8324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.1:55755
W20260928 16:45:21.406076 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:21.406386 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:21.406455 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:21.410805 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:21.410902 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:21.410996 8516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:45:21.415931 8516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:21.417047 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:21.418434 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:21.428532 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.428543 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.428516 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.428875 8516 server_base.cc:1034] running on GCE node
I20260928 16:45:21.429416 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:21.430188 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:21.431387 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790613921431354 us; error 51 us; skew 500 ppm
I20260928 16:45:21.433467 8516 webserver.cc:460] Webserver started at http://127.30.196.2:35343/ using document root <none> and password file <none>
I20260928 16:45:21.434062 8516 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:21.434182 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:21.434397 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:21.436589 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "e6f6c5e455ff45dba067ad39a0568c44"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "9a971dd22688650e4ab85b0bfda49c39"
server_key_iv: "a05111e1b372c7cc559c3390c5cdb28e"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.437170 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "e6f6c5e455ff45dba067ad39a0568c44"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "9a971dd22688650e4ab85b0bfda49c39"
server_key_iv: "a05111e1b372c7cc559c3390c5cdb28e"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.440855 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:45:21.443572 8530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.444962 8516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:21.445098 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "e6f6c5e455ff45dba067ad39a0568c44"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "9a971dd22688650e4ab85b0bfda49c39"
server_key_iv: "a05111e1b372c7cc559c3390c5cdb28e"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.445223 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:21.463567 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:21.464361 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:21.464579 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:21.465202 8516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:21.466157 8516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:21.466225 8516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.466301 8516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:21.466346 8516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.475207 8516 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:43439
I20260928 16:45:21.475209 8644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:43439 every 8 connection(s)
I20260928 16:45:21.476171 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:21.484365 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8516
I20260928 16:45:21.484508 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:45:21.484814 31504 external_mini_cluster.cc:1428] Setting key b0bd37f80ca24f2460927121d78eb613
I20260928 16:45:21.485415 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.477962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:47483 (local address 127.30.196.62:39727)
0928 16:45:21.478103 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.478130 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.479205 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 16:45:21.480633 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.480636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.480687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.480763 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.481681 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.482237 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.483349 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.483556 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.483645 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.484916 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:21.484921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:21.484924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:21.484939 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:21.484962 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:21.484973 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:21.485052 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:21.485227 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:45:21.485300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:21.486550 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.478366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39727 (local address 127.30.196.2:47483)
0928 16:45:21.479009 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:21.479050 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:21.480426 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:21.480924 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:21.480931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:21.481013 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:21.481533 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.481546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.482350 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.482352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.483235 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.483244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.483748 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.483751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.483892 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.484635 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:21.484639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:21.484691 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:21.484758 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:21.485055 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:21.485063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:21.485067 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:45:21.485196 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:45:21.485539 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260928 16:45:21.486981 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--builtin_ntp_servers=127.30.196.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:45:21.487655 8645 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39727
I20260928 16:45:21.487931 8645 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:21.488464 8645 heartbeater.cc:511] Master 127.30.196.62:39727 requested a full tablet report, sending...
I20260928 16:45:21.489522 8324 ts_manager.cc:194] Registered new tserver with Master: e6f6c5e455ff45dba067ad39a0568c44 (127.30.196.2:43439)
I20260928 16:45:21.490109 8324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.2:47483
W20260928 16:45:21.601568 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:21.601853 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:21.601890 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:21.606112 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:21.606178 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:21.606258 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:45:21.611245 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:21.612428 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:21.613847 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:21.623575 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.623590 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.623575 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.623860 8648 server_base.cc:1034] running on GCE node
I20260928 16:45:21.624419 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:21.625146 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:21.626374 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790613921626325 us; error 47 us; skew 500 ppm
I20260928 16:45:21.628446 8648 webserver.cc:460] Webserver started at http://127.30.196.3:39603/ using document root <none> and password file <none>
I20260928 16:45:21.629112 8648 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:21.629217 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:21.629479 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:21.631894 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "066c655e5644403197be3101952c76ae"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "db4476eada1c3e03ba9762f1a6b5b349"
server_key_iv: "2f02aa7d587f6d4483c1fbd57ca891dc"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.632588 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "066c655e5644403197be3101952c76ae"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "db4476eada1c3e03ba9762f1a6b5b349"
server_key_iv: "2f02aa7d587f6d4483c1fbd57ca891dc"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.636844 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:45:21.639622 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.640914 8648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:45:21.641059 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "066c655e5644403197be3101952c76ae"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "db4476eada1c3e03ba9762f1a6b5b349"
server_key_iv: "2f02aa7d587f6d4483c1fbd57ca891dc"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.641187 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:21.654289 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:21.655005 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:21.655210 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:21.655860 8648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:21.656828 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:21.656898 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.656968 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:21.657015 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.666080 8648 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:42615
I20260928 16:45:21.666113 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:42615 every 8 connection(s)
I20260928 16:45:21.667011 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:21.673916 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8648
I20260928 16:45:21.674045 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:45:21.674340 31504 external_mini_cluster.cc:1428] Setting key f16e5cc0f036142990bd48db8c9f9963
I20260928 16:45:21.676781 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.668892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:53037 (local address 127.30.196.62:39727)
0928 16:45:21.669065 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.669069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.670102 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 16:45:21.671525 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.671527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.671577 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.671659 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.672621 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.673251 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.674443 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.674706 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.674822 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.676213 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:21.676216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:21.676219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:21.676232 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:21.676255 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:21.676265 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:21.676345 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:21.676576 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:45:21.676638 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:45:21.677709 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.669305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39727 (local address 127.30.196.3:53037)
0928 16:45:21.669954 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:21.669994 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:21.671298 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:21.671874 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:21.671880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:21.671961 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:21.672476 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.672488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.673371 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.673374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.674317 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:21.674327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:21.674955 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:21.674960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:21.675170 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:21.675945 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:21.675948 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:21.675999 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:21.676066 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:21.676377 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:21.676394 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:21.676399 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:45:21.676564 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 16:45:21.676878 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260928 16:45:21.678874 8777 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39727
I20260928 16:45:21.679121 8777 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:21.679652 8777 heartbeater.cc:511] Master 127.30.196.62:39727 requested a full tablet report, sending...
I20260928 16:45:21.680720 8324 ts_manager.cc:194] Registered new tserver with Master: 066c655e5644403197be3101952c76ae (127.30.196.3:42615)
I20260928 16:45:21.681255 8324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.3:53037
I20260928 16:45:21.688524 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:45:21.691404 8363 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:21.690428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.30.196.62:39727)
0928 16:45:21.690580 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.690585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.690717 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:45:21.690873 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.690876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.690921 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.691005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.691276 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46350: BlockingRecv error: recv got EOF from 127.0.0.1:46350 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:45:21.691547 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8384
I20260928 16:45:21.697405 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8516
I20260928 16:45:21.702831 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8648
I20260928 16:45:21.708460 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8293
2026-09-28T16:45:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (908 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T16:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:21Z Disabled control of system clock
I20260928 16:45:21.737783 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35557
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:35557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:45:21.847745 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:21.848052 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:21.848090 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:21.852159 8788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:21.852221 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:21.852236 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:21.852253 8788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:21.852270 8788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:21.857165 8788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:35557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:21.858287 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:21.859685 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:21.868491 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:21.868521 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.868587 8788 server_base.cc:1034] running on GCE node
W20260928 16:45:21.868505 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:21.869391 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:21.870487 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:21.871690 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790613921871633 us; error 50 us; skew 500 ppm
I20260928 16:45:21.873647 8788 webserver.cc:460] Webserver started at http://127.30.196.62:41233/ using document root <none> and password file <none>
I20260928 16:45:21.874226 8788 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:21.874308 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:21.874562 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:21.876709 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "6bdcd38928d44637a73c20f7b57655f1"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "244ad11a4d62b03fcba38def7562a0ff"
server_key_iv: "9ccd8d09c0df65ce9a21b7936d3777e2"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.877270 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "6bdcd38928d44637a73c20f7b57655f1"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "244ad11a4d62b03fcba38def7562a0ff"
server_key_iv: "9ccd8d09c0df65ce9a21b7936d3777e2"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.881044 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:45:21.883648 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:21.884930 8788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:45:21.885040 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "6bdcd38928d44637a73c20f7b57655f1"
format_stamp: "Formatted at 2026-09-28 16:45:21 on dist-test-slave-5rg1"
server_key: "244ad11a4d62b03fcba38def7562a0ff"
server_key_iv: "9ccd8d09c0df65ce9a21b7936d3777e2"
server_key_version: "encryptionkey@0"
I20260928 16:45:21.885202 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:21.889932 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:21.890554 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:21.890770 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:21.897472 8788 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:35557
I20260928 16:45:21.897480 8854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:35557 every 8 connection(s)
I20260928 16:45:21.898419 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:21.901818 8855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:21.903383 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8788
I20260928 16:45:21.903504 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:45:21.903851 31504 external_mini_cluster.cc:1428] Setting key 0e60fb3067489a15e189a7c55f488ad5
I20260928 16:45:21.908223 8855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1: Bootstrap starting.
I20260928 16:45:21.909291 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:21.905305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.30.196.62:35557)
0928 16:45:21.905934 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:21.905949 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:21.905983 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:45:21.906631 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:21.906647 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:21.906980 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:21.907306 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:21.908098 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:21.908102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:21.908115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:21.908125 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:21.908176 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:21.908192 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:21.908336 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:21.908405 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:45:21.908638 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260928 16:45:21.910641 8855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:21.911449 8855 log.cc:824] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:21.913363 8855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1: No bootstrap required, opened a new log
I20260928 16:45:21.916731 8855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } }
I20260928 16:45:21.916967 8855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:21.917042 8855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdcd38928d44637a73c20f7b57655f1, State: Initialized, Role: FOLLOWER
I20260928 16:45:21.917475 8855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } }
I20260928 16:45:21.917649 8855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:21.917721 8855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:21.917819 8855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:21.918689 8855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } }
I20260928 16:45:21.919011 8855 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdcd38928d44637a73c20f7b57655f1; no voters:
I20260928 16:45:21.919317 8855 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:45:21.919512 8860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:21.919832 8860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [term 1 LEADER]: Becoming Leader. State: Replica: 6bdcd38928d44637a73c20f7b57655f1, State: Running, Role: LEADER
I20260928 16:45:21.920300 8860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } }
I20260928 16:45:21.920680 8855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:21.921911 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdcd38928d44637a73c20f7b57655f1. Latest consensus state: current_term: 1 leader_uuid: "6bdcd38928d44637a73c20f7b57655f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } } }
I20260928 16:45:21.921941 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdcd38928d44637a73c20f7b57655f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38928d44637a73c20f7b57655f1" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35557 } } }
I20260928 16:45:21.922045 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:21.922049 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:21.922422 8869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:21.925673 8869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:21.930902 8869 catalog_manager.cc:1347] Generated new cluster ID: c3e8176752f84304835236d67fae3b5f
I20260928 16:45:21.930982 8869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:21.938462 8869 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:45:21.938799 8869 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:21.947639 8869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdcd38928d44637a73c20f7b57655f1: Generated new TSK 0
I20260928 16:45:21.948468 8869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:45:21.958581 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--builtin_ntp_servers=127.30.196.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:45:22.073755 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:22.074059 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:22.074101 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:22.078233 8879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:22.078299 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:22.078359 8879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:45:22.083227 8879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:22.084368 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:22.085714 8879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:22.096032 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:22.096061 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:22.096148 8879 server_base.cc:1034] running on GCE node
W20260928 16:45:22.096035 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:22.096674 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:22.097388 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:22.098568 8879 hybrid_clock.cc:648] HybridClock initialized: now 1790613922098535 us; error 44 us; skew 500 ppm
I20260928 16:45:22.100564 8879 webserver.cc:460] Webserver started at http://127.30.196.1:43993/ using document root <none> and password file <none>
I20260928 16:45:22.101178 8879 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:22.101270 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:22.101500 8879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:22.103570 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "8ea3f971eb6e46e9b6e4e9a121ddd5eb"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "f3c5be514bef236121a063bc26304478"
server_key_iv: "9292142792760b0edc45cc62da0fada2"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.104266 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea3f971eb6e46e9b6e4e9a121ddd5eb"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "f3c5be514bef236121a063bc26304478"
server_key_iv: "9292142792760b0edc45cc62da0fada2"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.108174 8879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:45:22.111055 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.112445 8879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:22.112582 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "8ea3f971eb6e46e9b6e4e9a121ddd5eb"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "f3c5be514bef236121a063bc26304478"
server_key_iv: "9292142792760b0edc45cc62da0fada2"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.112710 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:22.117496 8879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:22.118113 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:22.118294 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:22.118832 8879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:22.119809 8879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:22.119872 8879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.119940 8879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:22.119979 8879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.129599 8879 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:46751
I20260928 16:45:22.129627 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:46751 every 8 connection(s)
I20260928 16:45:22.130522 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:22.135377 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 8879
I20260928 16:45:22.135507 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:45:22.135864 31504 external_mini_cluster.cc:1428] Setting key d9ef947b61c5094b0b8a49960c1a6e52
I20260928 16:45:22.136626 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.132368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:44307 (local address 127.30.196.62:35557)
0928 16:45:22.132503 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:22.132507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:22.133509 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 16:45:22.134752 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:22.134755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:22.134792 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:22.134865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:22.136075 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:22.136080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:22.136082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:22.136124 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:22.136149 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:22.136161 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:22.136260 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:22.136424 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:45:22.136489 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:45:22.137702 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.132688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:35557 (local address 127.30.196.1:44307)
0928 16:45:22.133361 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:22.133401 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:22.134569 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:22.134997 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:22.135003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:22.135085 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:22.135724 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:22.135726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:22.135766 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:22.135836 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:22.136273 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:22.136284 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:22.136289 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:45:22.136469 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 16:45:22.136711 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260928 16:45:22.138041 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--builtin_ntp_servers=127.30.196.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:45:22.139037 9007 heartbeater.cc:347] Connected to a master server at 127.30.196.62:35557
I20260928 16:45:22.139334 9007 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:22.139925 9007 heartbeater.cc:511] Master 127.30.196.62:35557 requested a full tablet report, sending...
I20260928 16:45:22.141645 8819 ts_manager.cc:194] Registered new tserver with Master: 8ea3f971eb6e46e9b6e4e9a121ddd5eb (127.30.196.1:46751)
W20260928 16:45:22.251466 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:22.251875 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:22.251962 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:22.256076 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:22.256172 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:22.256268 9010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:45:22.261339 9010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:22.262542 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:22.264139 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:22.274083 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:22.274180 9010 server_base.cc:1034] running on GCE node
W20260928 16:45:22.274091 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:22.274091 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:22.274890 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:22.275565 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:22.276782 9010 hybrid_clock.cc:648] HybridClock initialized: now 1790613922276758 us; error 41 us; skew 500 ppm
I20260928 16:45:22.278831 9010 webserver.cc:460] Webserver started at http://127.30.196.2:37713/ using document root <none> and password file <none>
I20260928 16:45:22.279467 9010 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:22.279567 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:22.279850 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:22.282064 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "e807d11383c24961a0db5a064d349a85"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "fee8aea61c9598afc5f78ea7e3364924"
server_key_iv: "12e1b9a19bdcac94a7d6e829750e75a1"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.282692 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "e807d11383c24961a0db5a064d349a85"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "fee8aea61c9598afc5f78ea7e3364924"
server_key_iv: "12e1b9a19bdcac94a7d6e829750e75a1"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.286679 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:45:22.289551 9024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.290904 9010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:45:22.291049 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "e807d11383c24961a0db5a064d349a85"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "fee8aea61c9598afc5f78ea7e3364924"
server_key_iv: "12e1b9a19bdcac94a7d6e829750e75a1"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.291195 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:22.296046 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:22.296940 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:22.297134 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:22.297696 9010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:22.298620 9010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:22.298666 9010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.298735 9010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:22.298768 9010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.307637 9010 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:37669
I20260928 16:45:22.307700 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:37669 every 8 connection(s)
I20260928 16:45:22.308604 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:22.314800 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.310484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:55479 (local address 127.30.196.62:35557)
0928 16:45:22.310630 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:22.310634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:22.311697 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 16:45:22.313084 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:22.313096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:22.313141 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:22.313223 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:22.314315 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:22.314319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:22.314322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:22.314335 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:22.314356 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:22.314365 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:22.314466 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:22.314608 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:45:22.314654 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:45:22.315359 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9010
I20260928 16:45:22.315506 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:45:22.315743 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.310872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:35557 (local address 127.30.196.2:55479)
0928 16:45:22.311520 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:22.311560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:22.312873 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:22.313377 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:22.313383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:22.313469 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:22.314067 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:22.314069 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:22.314111 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:22.314175 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:22.314458 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:22.314467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:22.314473 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:45:22.314648 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 16:45:22.314877 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260928 16:45:22.315917 31504 external_mini_cluster.cc:1428] Setting key d4c2848c36bfb285efdda48dc91c630e
I20260928 16:45:22.316983 9139 heartbeater.cc:347] Connected to a master server at 127.30.196.62:35557
I20260928 16:45:22.317214 9139 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:22.317662 9139 heartbeater.cc:511] Master 127.30.196.62:35557 requested a full tablet report, sending...
I20260928 16:45:22.317917 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--builtin_ntp_servers=127.30.196.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:45:22.318747 8819 ts_manager.cc:194] Registered new tserver with Master: e807d11383c24961a0db5a064d349a85 (127.30.196.2:37669)
W20260928 16:45:22.432579 9142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:22.432871 9142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:22.432914 9142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:22.437084 9142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:22.437146 9142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:22.437204 9142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:45:22.442055 9142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:35557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:22.443104 9142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:22.444573 9142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:22.454382 9147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:22.454479 9150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:22.454365 9148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:22.454744 9142 server_base.cc:1034] running on GCE node
I20260928 16:45:22.455456 9142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:22.456312 9142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:22.457460 9142 hybrid_clock.cc:648] HybridClock initialized: now 1790613922457427 us; error 49 us; skew 500 ppm
I20260928 16:45:22.459297 9142 webserver.cc:460] Webserver started at http://127.30.196.3:45199/ using document root <none> and password file <none>
I20260928 16:45:22.459952 9142 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:22.460018 9142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:22.460229 9142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:22.462283 9142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "461f26410445440383f0a63c8616bb6e"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "95c5c4a071ca3ff3aee84ced1c1631c5"
server_key_iv: "836586d8f9782b75da685155e1b31f36"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.462841 9142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "461f26410445440383f0a63c8616bb6e"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "95c5c4a071ca3ff3aee84ced1c1631c5"
server_key_iv: "836586d8f9782b75da685155e1b31f36"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.466461 9142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 16:45:22.468954 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.470165 9142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:45:22.470289 9142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "461f26410445440383f0a63c8616bb6e"
format_stamp: "Formatted at 2026-09-28 16:45:22 on dist-test-slave-5rg1"
server_key: "95c5c4a071ca3ff3aee84ced1c1631c5"
server_key_iv: "836586d8f9782b75da685155e1b31f36"
server_key_version: "encryptionkey@0"
I20260928 16:45:22.470404 9142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:22.474869 9142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:22.475445 9142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:22.475641 9142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:22.476212 9142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:22.477097 9142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:22.477144 9142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.477203 9142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:22.477238 9142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:22.485918 9142 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:32887
I20260928 16:45:22.485939 9270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:32887 every 8 connection(s)
I20260928 16:45:22.486816 9142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:22.493124 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.488733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:54739 (local address 127.30.196.62:35557)
0928 16:45:22.488899 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:22.488904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:22.490080 (+ 1176us) server_negotiation.cc:407] Connection header received
0928 16:45:22.491452 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:22.491455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:22.491491 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:22.491566 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:22.492736 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:22.492739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:22.492741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:45:22.492752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:22.492770 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:45:22.492778 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:22.492855 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:22.492976 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 16:45:22.493013 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:45:22.494023 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:22.489117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:35557 (local address 127.30.196.3:54739)
0928 16:45:22.489891 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:22.489952 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:22.491264 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:22.491788 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:22.491795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:22.491884 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:22.492456 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:45:22.492458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:22.492520 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:45:22.492595 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:22.492855 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:22.492862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:22.492867 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:45:22.493005 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:45:22.493197 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":107,"threads_started":1}
I20260928 16:45:22.494621 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9142
I20260928 16:45:22.494766 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:45:22.495106 31504 external_mini_cluster.cc:1428] Setting key bfefee8a5be015d984c266c7363c1bef
I20260928 16:45:22.495316 9271 heartbeater.cc:347] Connected to a master server at 127.30.196.62:35557
I20260928 16:45:22.495564 9271 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:22.496035 9271 heartbeater.cc:511] Master 127.30.196.62:35557 requested a full tablet report, sending...
I20260928 16:45:22.497120 8818 ts_manager.cc:194] Registered new tserver with Master: 461f26410445440383f0a63c8616bb6e (127.30.196.3:32887)
I20260928 16:45:22.509013 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:45:22.511816 8858 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:45:22.510858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.30.196.62:35557)
0928 16:45:22.511002 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:22.511006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:22.511194 (+ 188us) server_negotiation.cc:407] Connection header received
0928 16:45:22.511296 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:22.511298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:22.511332 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:22.511397 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:22.511636 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35086: BlockingRecv error: recv got EOF from 127.0.0.1:35086 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:22.511952 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8879
I20260928 16:45:22.519403 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9010
I20260928 16:45:22.526177 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9142
I20260928 16:45:22.532399 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 8788
2026-09-28T16:45:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (824 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:45:22 dist-test-slave-5rg1 krb5kdc[9282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:45:22 dist-test-slave-5rg1 krb5kdc[9282](info): set up 2 sockets
Sep 28 16:45:22 dist-test-slave-5rg1 krb5kdc[9282](info): commencing operation
krb5kdc: starting...
W20260928 16:45:23.597008 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:45:23 dist-test-slave-5rg1 krb5kdc[9282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:45:23.725067 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:43989
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:43989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:23.836341 9295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:23.836671 9295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:23.836751 9295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:23.840901 9295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:23.841122 9295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:23.841223 9295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:23.841274 9295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:23.841312 9295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:23.847259 9295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:43989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:23.848381 9295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:23.849404 9295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:23.857795 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:23.857789 9303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:23.857789 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:23.858026 9295 server_base.cc:1034] running on GCE node
I20260928 16:45:23.858680 9295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:23.859823 9295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:23.861024 9295 hybrid_clock.cc:648] HybridClock initialized: now 1790613923861003 us; error 42 us; skew 500 ppm
Sep 28 16:45:23 dist-test-slave-5rg1 krb5kdc[9282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613923, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:23.868479 9295 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:45:23.869460 9295 webserver.cc:460] Webserver started at http://127.30.196.62:44689/ using document root <none> and password file <none>
I20260928 16:45:23.870010 9295 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:23.870070 9295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:23.870250 9295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:23.872460 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "a2240c5f771c45fb86546efc35bf3df9"
format_stamp: "Formatted at 2026-09-28 16:45:23 on dist-test-slave-5rg1"
server_key: "ceea43f7e3aaad4590df4fb1cbb1d945"
server_key_iv: "12108cc4ea3c9a2c5adee95a777b732a"
server_key_version: "encryptionkey@0"
I20260928 16:45:23.872987 9295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "a2240c5f771c45fb86546efc35bf3df9"
format_stamp: "Formatted at 2026-09-28 16:45:23 on dist-test-slave-5rg1"
server_key: "ceea43f7e3aaad4590df4fb1cbb1d945"
server_key_iv: "12108cc4ea3c9a2c5adee95a777b732a"
server_key_version: "encryptionkey@0"
I20260928 16:45:23.877049 9295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:45:23.879750 9310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:23.881047 9295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:45:23.881186 9295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "a2240c5f771c45fb86546efc35bf3df9"
format_stamp: "Formatted at 2026-09-28 16:45:23 on dist-test-slave-5rg1"
server_key: "ceea43f7e3aaad4590df4fb1cbb1d945"
server_key_iv: "12108cc4ea3c9a2c5adee95a777b732a"
server_key_version: "encryptionkey@0"
I20260928 16:45:23.881284 9295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:23.895494 9295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:23.898751 9295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:23.898910 9295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:23.905850 9295 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:43989
I20260928 16:45:23.906173 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:43989 every 8 connection(s)
I20260928 16:45:23.906823 9295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:23.909797 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:23.911585 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9295
I20260928 16:45:23.911741 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:45:23.912010 31504 external_mini_cluster.cc:1428] Setting key e4c069ddc980876fbaf5659be19bf36f
I20260928 16:45:23.916759 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: Bootstrap starting.
I20260928 16:45:23.919418 9363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:23.920228 9363 log.cc:824] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:45:23 dist-test-slave-5rg1 krb5kdc[9282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:23.922344 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: No bootstrap required, opened a new log
I20260928 16:45:23.927325 9363 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } }
I20260928 16:45:23.927559 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:23.913425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.30.196.62:43989)
0928 16:45:23.913850 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:23.913869 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:23.913929 (+ 60us) server_negotiation.cc:407] Connection header received
0928 16:45:23.914807 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:23.914832 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:23.915259 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:23.915643 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:23.916354 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:23.917770 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:23.918595 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:23.919263 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:23.919435 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:23.921980 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:23.921988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:23.922006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:23.922038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:23.924116 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:23.924618 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:23.924624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:23.924630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:23.924793 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:23.925149 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:23.925155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:23.925157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:23.925717 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:23.925906 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:23.926214 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 16:45:23.926546 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260928 16:45:23.927613 9363 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:23.927659 9363 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2240c5f771c45fb86546efc35bf3df9, State: Initialized, Role: FOLLOWER
I20260928 16:45:23.928167 9363 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } }
I20260928 16:45:23.928295 9363 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:23.928378 9363 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:23.928478 9363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:23.929731 9363 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } }
I20260928 16:45:23.930133 9363 leader_election.cc:302] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2240c5f771c45fb86546efc35bf3df9; no voters:
I20260928 16:45:23.930474 9363 leader_election.cc:288] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:45:23.930612 9368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:23.930847 9368 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [term 1 LEADER]: Becoming Leader. State: Replica: a2240c5f771c45fb86546efc35bf3df9, State: Running, Role: LEADER
I20260928 16:45:23.931188 9368 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } }
I20260928 16:45:23.931770 9363 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:23.932861 9369 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2240c5f771c45fb86546efc35bf3df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } } }
I20260928 16:45:23.932998 9369 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:23.933362 9370 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2240c5f771c45fb86546efc35bf3df9. Latest consensus state: current_term: 1 leader_uuid: "a2240c5f771c45fb86546efc35bf3df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2240c5f771c45fb86546efc35bf3df9" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43989 } } }
I20260928 16:45:23.933377 9377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:23.933589 9370 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:23.936573 9377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:23.942165 9377 catalog_manager.cc:1347] Generated new cluster ID: ff1953ddddd14ae4811203ca9c0d8b8e
I20260928 16:45:23.942240 9377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:23.959237 9377 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:45:23.960314 9377 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:23.969085 9377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: Generated new TSK 0
I20260928 16:45:23.970300 9377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:45:24.011619 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:24.126775 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:24.127089 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:24.127134 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:24.131459 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:24.131538 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:24.131644 9389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:45:24.137596 9389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:24.138827 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:24.139951 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:24.150281 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:24.150295 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:24.150291 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:24.151230 9389 server_base.cc:1034] running on GCE node
I20260928 16:45:24.151721 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:24.152617 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:24.153820 9389 hybrid_clock.cc:648] HybridClock initialized: now 1790613924153728 us; error 115 us; skew 500 ppm
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:24.157079 9389 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:45:24.158329 9389 webserver.cc:460] Webserver started at http://127.30.196.1:44883/ using document root <none> and password file <none>
I20260928 16:45:24.159030 9389 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:24.159101 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:24.159343 9389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:24.162016 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "e840c995952340fe8b40d44db6fb2cfb"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "1fca0e5ce62a99791bcf5bec921654d0"
server_key_iv: "0d1158588685acc34ba72a0b2593c2b0"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.162624 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "e840c995952340fe8b40d44db6fb2cfb"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "1fca0e5ce62a99791bcf5bec921654d0"
server_key_iv: "0d1158588685acc34ba72a0b2593c2b0"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.166653 9389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:45:24.169353 9404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.170776 9389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:45:24.170984 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "e840c995952340fe8b40d44db6fb2cfb"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "1fca0e5ce62a99791bcf5bec921654d0"
server_key_iv: "0d1158588685acc34ba72a0b2593c2b0"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.171119 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:24.199064 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:24.202522 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:24.202746 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:24.203440 9389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:24.204384 9389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:24.204456 9389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.204526 9389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:24.204559 9389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.213577 9389 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:37717
I20260928 16:45:24.213598 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:37717 every 8 connection(s)
I20260928 16:45:24.214481 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:24.219030 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9389
I20260928 16:45:24.219127 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:45:24.219404 31504 external_mini_cluster.cc:1428] Setting key 35e02476cc00b35331e571c6b83c7efa
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:24.229185 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.216314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:44673 (local address 127.30.196.62:43989)
0928 16:45:24.216457 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.216460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.217277 (+ 817us) server_negotiation.cc:407] Connection header received
0928 16:45:24.218758 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.218762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.218822 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.218928 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:24.220280 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.220835 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.221981 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.222217 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.222330 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.224908 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:24.224912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:24.224914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:24.224940 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:24.226739 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.227303 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.227306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.227310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.227404 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.227786 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.227790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.227792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.228048 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:24.228179 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.228802 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.228982 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:45:24.230023 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.216675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:43989 (local address 127.30.196.1:44673)
0928 16:45:24.217128 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.217171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.218543 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.219130 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.219141 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.219585 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:24.220135 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.220150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.220965 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.220968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.221853 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.221863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.222380 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.222384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.222560 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.223196 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:24.223213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:24.224720 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:24.226891 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.226895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.226910 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.227197 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.227500 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.227502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.227503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.227644 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.228218 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:24.228224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:24.228497 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.228685 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.229030 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260928 16:45:24.231410 9519 heartbeater.cc:347] Connected to a master server at 127.30.196.62:43989
I20260928 16:45:24.231702 9519 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:24.232257 9519 heartbeater.cc:511] Master 127.30.196.62:43989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:45:24.234115 9323 ts_manager.cc:194] Registered new tserver with Master: e840c995952340fe8b40d44db6fb2cfb (127.30.196.1:37717)
I20260928 16:45:24.235361 9323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:44673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:45:24.251857 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:24.367743 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:24.368101 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:24.368188 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:24.372380 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:24.372444 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:24.372525 9524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:45:24.378324 9524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:24.379477 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:24.380576 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:24.390308 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:24.390426 9524 server_base.cc:1034] running on GCE node
W20260928 16:45:24.390365 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:24.390360 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:24.390938 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:24.391635 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:24.392860 9524 hybrid_clock.cc:648] HybridClock initialized: now 1790613924392822 us; error 47 us; skew 500 ppm
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:24.395591 9524 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:45:24.396512 9524 webserver.cc:460] Webserver started at http://127.30.196.2:41079/ using document root <none> and password file <none>
I20260928 16:45:24.397091 9524 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:24.397182 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:24.397408 9524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:24.399377 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "6669466e78834e7aac47570b27535d16"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "3be9c85e03ec71e90a7a3209b584cc78"
server_key_iv: "d3ce772c00455c9abddcaaa602ce36ed"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.399959 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "6669466e78834e7aac47570b27535d16"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "3be9c85e03ec71e90a7a3209b584cc78"
server_key_iv: "d3ce772c00455c9abddcaaa602ce36ed"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.403522 9524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:45:24.405879 9539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.406984 9524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:45:24.407116 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "6669466e78834e7aac47570b27535d16"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "3be9c85e03ec71e90a7a3209b584cc78"
server_key_iv: "d3ce772c00455c9abddcaaa602ce36ed"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.407222 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:24.421267 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:24.424183 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:24.424384 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:24.424952 9524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:24.425776 9524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:24.425844 9524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.425910 9524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:24.425958 9524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.435012 9524 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38153
I20260928 16:45:24.435070 9653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38153 every 8 connection(s)
I20260928 16:45:24.436034 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:24.439141 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9524
I20260928 16:45:24.439267 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:45:24.439532 31504 external_mini_cluster.cc:1428] Setting key 11c3e27429c65bc3205018239faee652
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:45:24.449494 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.437814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:49927 (local address 127.30.196.62:43989)
0928 16:45:24.437944 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.437947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.438751 (+ 804us) server_negotiation.cc:407] Connection header received
0928 16:45:24.440329 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.440332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.440383 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.440483 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:24.441676 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.442193 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.443291 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.443508 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.443624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.445947 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:24.445956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:24.445959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:24.445981 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:24.447397 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.447967 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.447972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.447974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.448050 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.448383 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.448386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.448387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.448606 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:24.448709 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.449198 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.449321 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:45:24.450328 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.438184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:43989 (local address 127.30.196.2:49927)
0928 16:45:24.438615 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.438655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.440126 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.440647 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.440653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.441076 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:24.441545 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.441557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.442310 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.442313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.443172 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.443181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.443650 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.443655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.443855 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.444422 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:24.444438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:24.445792 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:24.447517 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.447523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.447537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.447867 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.448146 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.448149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.448151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.448285 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.448707 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:24.448714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:24.448965 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.449145 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.449457 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:45:24.451537 9654 heartbeater.cc:347] Connected to a master server at 127.30.196.62:43989
I20260928 16:45:24.451793 9654 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:24.452267 9654 heartbeater.cc:511] Master 127.30.196.62:43989 requested a full tablet report, sending...
I20260928 16:45:24.453253 9323 ts_manager.cc:194] Registered new tserver with Master: 6669466e78834e7aac47570b27535d16 (127.30.196.2:38153)
I20260928 16:45:24.453820 9323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:49927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:45:24.467921 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:24.583003 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:24.583292 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:24.583338 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:24.587906 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:24.587975 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:24.588037 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:45:24.593514 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:45:24.594574 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:24.595592 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:24.605232 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:24.605312 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:24.605353 9659 server_base.cc:1034] running on GCE node
W20260928 16:45:24.605315 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:24.606035 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:24.606720 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:24.607896 9659 hybrid_clock.cc:648] HybridClock initialized: now 1790613924607886 us; error 70 us; skew 500 ppm
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:24.610707 9659 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:45:24.611653 9659 webserver.cc:460] Webserver started at http://127.30.196.3:34817/ using document root <none> and password file <none>
I20260928 16:45:24.612260 9659 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:24.612350 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:24.612576 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:45:24.614568 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "f099e601808441399efe66c879bf8066"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "d5cc2e9503c813d64a39e752b2e92672"
server_key_iv: "3f9d55442daa9b58a27bfde7fb0068de"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.615139 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "f099e601808441399efe66c879bf8066"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "d5cc2e9503c813d64a39e752b2e92672"
server_key_iv: "3f9d55442daa9b58a27bfde7fb0068de"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.618794 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 16:45:24.621227 9674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.622355 9659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:45:24.622510 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "f099e601808441399efe66c879bf8066"
format_stamp: "Formatted at 2026-09-28 16:45:24 on dist-test-slave-5rg1"
server_key: "d5cc2e9503c813d64a39e752b2e92672"
server_key_iv: "3f9d55442daa9b58a27bfde7fb0068de"
server_key_version: "encryptionkey@0"
I20260928 16:45:24.622622 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:24.640996 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:24.644541 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:24.644767 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:24.645390 9659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:24.646253 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:24.646322 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.646389 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:24.646450 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:24.655959 9659 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:35385
I20260928 16:45:24.656005 9790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:35385 every 8 connection(s)
I20260928 16:45:24.656909 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:24.665017 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 9659
I20260928 16:45:24.665191 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:45:24.665577 31504 external_mini_cluster.cc:1428] Setting key ffe604bf29e239fc6013cd7898c30c58
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613924, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:45:24.670869 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.658787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:34043 (local address 127.30.196.62:43989)
0928 16:45:24.658926 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.658930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.659843 (+ 913us) server_negotiation.cc:407] Connection header received
0928 16:45:24.661224 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.661228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.661290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.661383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:24.662563 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.663104 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.664243 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.664476 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.664585 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.667513 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:24.667516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:24.667519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:24.667548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:24.669000 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.669495 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.669497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.669499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.669567 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.669865 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.669867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.669869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.670037 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:24.670115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.670656 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.670752 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:45:24.671805 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.659188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:43989 (local address 127.30.196.3:34043)
0928 16:45:24.659654 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.659744 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.660984 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.661555 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.661561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.661932 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:24.662413 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.662425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.663227 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.663230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.664117 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.664126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.664636 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.664641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.664822 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.665501 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:24.665517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:24.667353 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:24.669122 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.669128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.669140 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.669407 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.669656 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:24.669658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:24.669660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:24.669791 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:24.670123 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:24.670128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:24.670378 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.670602 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.670870 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260928 16:45:24.672911 9791 heartbeater.cc:347] Connected to a master server at 127.30.196.62:43989
I20260928 16:45:24.673156 9791 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:24.673666 9791 heartbeater.cc:511] Master 127.30.196.62:43989 requested a full tablet report, sending...
I20260928 16:45:24.674719 9323 ts_manager.cc:194] Registered new tserver with Master: f099e601808441399efe66c879bf8066 (127.30.196.3:35385)
I20260928 16:45:24.675365 9323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:34043
I20260928 16:45:24.679479 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:45:24.688640 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.681412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.30.196.62:43989)
0928 16:45:24.681540 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.681549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.681703 (+ 154us) server_negotiation.cc:407] Connection header received
0928 16:45:24.681802 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.681804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.681840 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.681909 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:24.682645 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.683160 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.684076 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.684288 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.684377 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.685728 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:24.685731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:24.685734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:24.685756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:24.687032 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.687462 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.687465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.687466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.687547 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:24.687867 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:24.687870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:24.687871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:24.688108 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:24.688223 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.688410 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.688523 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:24.691346 9323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:45:24.694499 9323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:24.709693 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.703196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:35385 (local address 127.0.0.1:39590)
0928 16:45:24.704283 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.704295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.704427 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.704930 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.704935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.704953 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:24.705047 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.705052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.706488 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.706491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.707968 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.707980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.709236 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.709239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.709328 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.709360 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.709467 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.709560 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":257,"thread_start_us":118,"threads_started":1}
I20260928 16:45:24.709796 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.703390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.30.196.3:35385)
0928 16:45:24.703939 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.703953 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.704365 (+ 412us) server_negotiation.cc:407] Connection header received
0928 16:45:24.704623 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.704628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.704805 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.704964 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:24.705140 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.706360 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.708104 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.708952 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.709063 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.709210 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.709519 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.709614 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":132,"threads_started":1}
I20260928 16:45:24.710808 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.704306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38153 (local address 127.0.0.1:58734)
0928 16:45:24.704697 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.704710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.704775 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.705562 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.705565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.705584 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:24.705667 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.705672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.707064 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.707068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.707979 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.707990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.710359 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.710362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.710475 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.710498 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.710573 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.710656 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260928 16:45:24.710808 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.703124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:37717 (local address 127.0.0.1:59190)
0928 16:45:24.703715 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.703765 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.704045 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.704698 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.704704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.704734 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:24.704895 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.704905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.706340 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.706346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.707968 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.707980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.710440 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.710443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.710513 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.710547 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.710634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.710686 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":69,"thread_start_us":92,"threads_started":1}
I20260928 16:45:24.711151 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.703824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.30.196.1:37717)
0928 16:45:24.704168 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.704185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.704208 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:45:24.704283 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.704289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.704525 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.705058 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:24.705215 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.706214 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.709282 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.710317 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.710429 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.710555 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.710879 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.710996 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":141,"threads_started":1}
I20260928 16:45:24.711328 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.704724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.30.196.2:38153)
0928 16:45:24.705148 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.705159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.705179 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:45:24.705245 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.705250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.705409 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.705554 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:24.705788 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.706916 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.708151 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.708952 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.709063 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.709210 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.711066 (+ 1856us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.711169 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":139,"threads_started":1}
I20260928 16:45:24.713187 9724 tablet_service.cc:1467] Processing CreateTablet for tablet 0f027d80ae58442cb09be4adc49cee99 (DEFAULT_TABLE table=test-table [id=337eca41445c477b9d86dc779f665558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:24.713935 9587 tablet_service.cc:1467] Processing CreateTablet for tablet 0f027d80ae58442cb09be4adc49cee99 (DEFAULT_TABLE table=test-table [id=337eca41445c477b9d86dc779f665558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:24.714371 9724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f027d80ae58442cb09be4adc49cee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:24.714071 9453 tablet_service.cc:1467] Processing CreateTablet for tablet 0f027d80ae58442cb09be4adc49cee99 (DEFAULT_TABLE table=test-table [id=337eca41445c477b9d86dc779f665558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:24.714888 9587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f027d80ae58442cb09be4adc49cee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:24.714950 9453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f027d80ae58442cb09be4adc49cee99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:24.722102 9810 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Bootstrap starting.
I20260928 16:45:24.722311 9808 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: Bootstrap starting.
I20260928 16:45:24.722754 9809 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Bootstrap starting.
I20260928 16:45:24.724529 9808 tablet_bootstrap.cc:654] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:24.724865 9809 tablet_bootstrap.cc:654] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:24.724956 9810 tablet_bootstrap.cc:654] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:24.725611 9809 log.cc:824] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:24.725777 9810 log.cc:824] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:24.725777 9808 log.cc:824] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:24.727607 9809 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: No bootstrap required, opened a new log
I20260928 16:45:24.727829 9809 ts_tablet_manager.cc:1397] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:45:24.728106 9808 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: No bootstrap required, opened a new log
I20260928 16:45:24.728284 9808 ts_tablet_manager.cc:1397] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:45:24.730726 9809 raft_consensus.cc:348] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.730979 9809 raft_consensus.cc:374] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:24.731042 9809 raft_consensus.cc:729] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f099e601808441399efe66c879bf8066, State: Initialized, Role: FOLLOWER
I20260928 16:45:24.731172 9808 raft_consensus.cc:348] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.731441 9808 raft_consensus.cc:374] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:24.731498 9808 raft_consensus.cc:729] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e840c995952340fe8b40d44db6fb2cfb, State: Initialized, Role: FOLLOWER
I20260928 16:45:24.731571 9809 consensus_queue.cc:260] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.731954 9808 consensus_queue.cc:260] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.732461 9791 heartbeater.cc:503] Master 127.30.196.62:43989 was elected leader, sending a full tablet report...
I20260928 16:45:24.732820 9809 ts_tablet_manager.cc:1428] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:45:24.732865 9808 ts_tablet_manager.cc:1428] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:45:24.733165 9519 heartbeater.cc:503] Master 127.30.196.62:43989 was elected leader, sending a full tablet report...
I20260928 16:45:24.733680 9810 tablet_bootstrap.cc:492] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: No bootstrap required, opened a new log
I20260928 16:45:24.733868 9810 ts_tablet_manager.cc:1397] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260928 16:45:24.737182 9810 raft_consensus.cc:348] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.737489 9810 raft_consensus.cc:374] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:24.737561 9810 raft_consensus.cc:729] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6669466e78834e7aac47570b27535d16, State: Initialized, Role: FOLLOWER
I20260928 16:45:24.737996 9810 consensus_queue.cc:260] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.738651 9654 heartbeater.cc:503] Master 127.30.196.62:43989 was elected leader, sending a full tablet report...
I20260928 16:45:24.738890 9810 ts_tablet_manager.cc:1428] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:45:24.754251 9815 raft_consensus.cc:482] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:24.754487 9815 raft_consensus.cc:504] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.755642 9815 leader_election.cc:288] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f099e601808441399efe66c879bf8066 (127.30.196.3:35385), 6669466e78834e7aac47570b27535d16 (127.30.196.2:38153)
I20260928 16:45:24.759775 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.756066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:47727 (local address 127.30.196.2:38153)
0928 16:45:24.756207 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.756213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.756431 (+ 218us) server_negotiation.cc:407] Connection header received
0928 16:45:24.756608 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.756611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.756663 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.756746 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:24.757066 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.757589 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.758758 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.759182 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.759314 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.759368 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.759544 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.759614 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:45:24.759790 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.755909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38153 (local address 127.30.196.1:47727)
0928 16:45:24.756338 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.756350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.756487 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.756758 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.756761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.756776 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:24.756944 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.756952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.757736 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.757741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.758634 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.758643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.759309 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.759313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.759401 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.759423 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.759527 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.759601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":65,"thread_start_us":170,"threads_started":1}
I20260928 16:45:24.759789 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.756121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:32773 (local address 127.30.196.3:35385)
0928 16:45:24.756259 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.756264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.756545 (+ 281us) server_negotiation.cc:407] Connection header received
0928 16:45:24.756689 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.756692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.756740 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.756818 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:24.757086 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.757607 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.758760 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.759231 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.759348 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.759403 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.759576 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.759647 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:45:24.759807 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.755929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:35385 (local address 127.30.196.1:32773)
0928 16:45:24.756457 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.756467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:24.756581 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:24.756831 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:24.756834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:24.756850 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:24.756985 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.756991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.757718 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.757722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.758633 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:24.758642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.759343 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.759347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:24.759449 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.759471 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:45:24.759561 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:45:24.759632 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":38,"thread_start_us":148,"threads_started":1}
I20260928 16:45:24.760468 9745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f027d80ae58442cb09be4adc49cee99" candidate_uuid: "e840c995952340fe8b40d44db6fb2cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f099e601808441399efe66c879bf8066" is_pre_election: true
I20260928 16:45:24.760465 9608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f027d80ae58442cb09be4adc49cee99" candidate_uuid: "e840c995952340fe8b40d44db6fb2cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6669466e78834e7aac47570b27535d16" is_pre_election: true
I20260928 16:45:24.760788 9745 raft_consensus.cc:2454] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e840c995952340fe8b40d44db6fb2cfb in term 0.
I20260928 16:45:24.760792 9608 raft_consensus.cc:2454] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e840c995952340fe8b40d44db6fb2cfb in term 0.
I20260928 16:45:24.761291 9407 leader_election.cc:302] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6669466e78834e7aac47570b27535d16, e840c995952340fe8b40d44db6fb2cfb, f099e601808441399efe66c879bf8066; no voters:
I20260928 16:45:24.761559 9815 raft_consensus.cc:2781] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:24.761649 9815 raft_consensus.cc:482] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:24.761710 9815 raft_consensus.cc:3037] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:24.762734 9815 raft_consensus.cc:504] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.763089 9815 leader_election.cc:288] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [CANDIDATE]: Term 1 election: Requested vote from peers f099e601808441399efe66c879bf8066 (127.30.196.3:35385), 6669466e78834e7aac47570b27535d16 (127.30.196.2:38153)
I20260928 16:45:24.763433 9745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f027d80ae58442cb09be4adc49cee99" candidate_uuid: "e840c995952340fe8b40d44db6fb2cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f099e601808441399efe66c879bf8066"
I20260928 16:45:24.763513 9608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f027d80ae58442cb09be4adc49cee99" candidate_uuid: "e840c995952340fe8b40d44db6fb2cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6669466e78834e7aac47570b27535d16"
I20260928 16:45:24.763572 9745 raft_consensus.cc:3037] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:24.763631 9608 raft_consensus.cc:3037] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:24.764991 9745 raft_consensus.cc:2454] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e840c995952340fe8b40d44db6fb2cfb in term 1.
I20260928 16:45:24.765067 9608 raft_consensus.cc:2454] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e840c995952340fe8b40d44db6fb2cfb in term 1.
I20260928 16:45:24.765359 9406 leader_election.cc:302] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6669466e78834e7aac47570b27535d16, e840c995952340fe8b40d44db6fb2cfb; no voters:
I20260928 16:45:24.765655 9815 raft_consensus.cc:2781] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:24.765890 9815 raft_consensus.cc:686] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 1 LEADER]: Becoming Leader. State: Replica: e840c995952340fe8b40d44db6fb2cfb, State: Running, Role: LEADER
I20260928 16:45:24.766255 9815 consensus_queue.cc:237] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } }
I20260928 16:45:24.769420 9325 catalog_manager.cc:5495] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb reported cstate change: term changed from 0 to 1, leader changed from <none> to e840c995952340fe8b40d44db6fb2cfb (127.30.196.1). New cstate: current_term: 1 leader_uuid: "e840c995952340fe8b40d44db6fb2cfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e840c995952340fe8b40d44db6fb2cfb" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260928 16:45:24.789944 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.786050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.30.196.1:37717)
0928 16:45:24.786178 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:24.786183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:24.786273 (+ 90us) server_negotiation.cc:407] Connection header received
0928 16:45:24.786486 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:24.786489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:24.786550 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:24.786664 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:24.786920 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.787421 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.788421 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:24.788646 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:24.788757 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:24.789087 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:24.789528 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:24.789662 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:24.789725 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 16:45:24.789789 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:45:24.795890 9608 raft_consensus.cc:1261] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 1 FOLLOWER]: Refusing update from remote peer e840c995952340fe8b40d44db6fb2cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:24.795890 9745 raft_consensus.cc:1261] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 1 FOLLOWER]: Refusing update from remote peer e840c995952340fe8b40d44db6fb2cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:24.796648 9819 consensus_queue.cc:1035] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f099e601808441399efe66c879bf8066" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:24.796842 9815 consensus_queue.cc:1035] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6669466e78834e7aac47570b27535d16" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:24.805634 9823 mvcc.cc:205] Tried to move back new op lower bound from 7334354635956477952 to 7334354635845423104. Current Snapshot: MvccSnapshot[applied={T|T < 7334354635956477952}]
I20260928 16:45:24.804904 9824 mvcc.cc:205] Tried to move back new op lower bound from 7334354635956477952 to 7334354635845423104. Current Snapshot: MvccSnapshot[applied={T|T < 7334354635956477952}]
I20260928 16:45:24.813158 9325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:24.813321 9325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:24.815783 9325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2240c5f771c45fb86546efc35bf3df9: Sending DeleteTablet for 3 replicas of tablet 0f027d80ae58442cb09be4adc49cee99
I20260928 16:45:24.816699 9724 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f027d80ae58442cb09be4adc49cee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:24 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39590
I20260928 16:45:24.816702 9453 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f027d80ae58442cb09be4adc49cee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:24 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59190
I20260928 16:45:24.817265 9832 tablet_replica.cc:330] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb: stopping tablet replica
I20260928 16:45:24.817247 9587 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f027d80ae58442cb09be4adc49cee99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:24 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:58734
I20260928 16:45:24.817622 9832 raft_consensus.cc:2229] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:24.817751 9834 tablet_replica.cc:330] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: stopping tablet replica
I20260928 16:45:24.818044 9832 raft_consensus.cc:2258] T 0f027d80ae58442cb09be4adc49cee99 P e840c995952340fe8b40d44db6fb2cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:24.818079 9834 raft_consensus.cc:2229] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:24.818410 9833 tablet_replica.cc:330] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: stopping tablet replica
I20260928 16:45:24.818687 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9389
I20260928 16:45:24.818689 9833 raft_consensus.cc:2229] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:24.819454 9833 raft_consensus.cc:2258] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:24.821136 9833 ts_tablet_manager.cc:1905] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:24.821208 9834 raft_consensus.cc:2258] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:24.822693 9834 ts_tablet_manager.cc:1905] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:24.823515 9833 ts_tablet_manager.cc:1918] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:24.823630 9833 log.cc:1196] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/0f027d80ae58442cb09be4adc49cee99
I20260928 16:45:24.823982 9833 ts_tablet_manager.cc:1939] T 0f027d80ae58442cb09be4adc49cee99 P f099e601808441399efe66c879bf8066: Deleting consensus metadata
I20260928 16:45:24.825153 9313 catalog_manager.cc:4841] TS f099e601808441399efe66c879bf8066 (127.30.196.3:35385): tablet 0f027d80ae58442cb09be4adc49cee99 (table test-table [id=337eca41445c477b9d86dc779f665558]) successfully deleted
I20260928 16:45:24.826138 9834 ts_tablet_manager.cc:1918] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:24.826247 9834 log.cc:1196] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/0f027d80ae58442cb09be4adc49cee99
I20260928 16:45:24.826581 9834 ts_tablet_manager.cc:1939] T 0f027d80ae58442cb09be4adc49cee99 P 6669466e78834e7aac47570b27535d16: Deleting consensus metadata
W20260928 16:45:24.826962 9314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:37717 (error 108)
I20260928 16:45:24.827143 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9524
I20260928 16:45:24.828758 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.828486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:37717 (local address 127.0.0.1:59210)
0928 16:45:24.828614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.828685 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:45:24.828987 9314 catalog_manager.cc:4584] TS e840c995952340fe8b40d44db6fb2cfb (127.30.196.1:37717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f027d80ae58442cb09be4adc49cee99: Network error: Client connection negotiation failed: client connection to 127.30.196.1:37717: connect: Connection refused (error 111)
I20260928 16:45:24.832792 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9659
I20260928 16:45:24.834046 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:24.833022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38153 (local address 127.0.0.1:58748)
0928 16:45:24.833860 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:24.833931 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":734}
I20260928 16:45:24.838426 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 9295
2026-09-28T16:45:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2307 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9838](info): set up 2 sockets
Sep 28 16:45:24 dist-test-slave-5rg1 krb5kdc[9838](info): commencing operation
krb5kdc: starting...
W20260928 16:45:25.885175 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:45:25 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangeradmin_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangeradmin_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangerlookup_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangerlookup_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
I20260928 16:45:26.066941 31504 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6WsIs2/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/postgres -l logfile start
2026-09-28 16:45:28.880 UTC [9868] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:45:28.880 UTC [9868] LOG: listening on IPv4 address "127.30.196.20", port 36123
2026-09-28 16:45:28.885 UTC [9868] LOG: listening on Unix socket "/tmp/.s.PGSQL.36123"
2026-09-28 16:45:28.907 UTC [9871] LOG: database system was shut down at 2026-09-28 16:45:27 UTC
2026-09-28 16:45:28.911 UTC [9868] LOG: database system is ready to accept connections
I20260928 16:45:29.876999 31504 mini_postgres.cc:96] Postgres bound to 36123
2026-09-28 16:45:29.883 UTC [9879] FATAL: database "slave" does not exist
127.30.196.20:36123 - accepting connections
I20260928 16:45:29.884583 31504 mini_ranger.cc:161] Starting Ranger...
I20260928 16:45:29.901645 31504 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:45:30.489571 31504 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:45:30.489650 31504 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin
2026-09-28 16:45:30,772 [I] DB FLAVOR :POSTGRES
2026-09-28 16:45:30,773 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:45:30,773 [I] Checking connection..
2026-09-28 16:45:30,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:45:31,048 [I] Checking connection passed.
2026-09-28 16:45:31,048 [I] --------- Verifying version history table ---------
2026-09-28 16:45:31,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:31,290 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:45:31,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:31,535 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:45:31,536 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:45:31,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:45:31.775 UTC [9943] WARNING: there is no transaction in progress
2026-09-28 16:45:31,782 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:45:31,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:32,051 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:45:32,051 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:45:32,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:45:32,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:45:32,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5rg1.c.gcp-upstream.internal','N') ;"
2026-09-28 16:45:32,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:45:33,097 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:45:33,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:45:33,351 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:45:33,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:45:33,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:45:33,833 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:45:33,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:45:34.931 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:34.949 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:34.968 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:34.987 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.006 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.028 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.090 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.098 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.114 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.124 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.135 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.151 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.162 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.169 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.177 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35.185 UTC [10051] WARNING: there is no transaction in progress
2026-09-28 16:45:35,920 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:45:35,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:45:36,184 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:45:36,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:45:36,447 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:45:36,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:45:36,727 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:45:36,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:45:36,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:45:37,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5rg1.c.gcp-upstream.internal';"
2026-09-28 16:45:37,501 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:45:37,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:45:37,771 [I] DB_PATCHES have already been applied
I20260928 16:45:37.777899 31504 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6WsIs2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:45:37.777983 31504 mini_ranger.cc:193] Using host: 127.30.196.20
I20260928 16:45:37.781840 31504 mini_ranger.cc:227] Ranger admin URL: http://127.30.196.20:43799
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:45:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:45:38.927628 31504 mini_ranger.cc:160] Time spent starting Ranger: real 9.043s user 0.000s sys 0.011s
Sep 28, 2026 4:45:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:45:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:45:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:45:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:46:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:46:00.933931 31504 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangerkms_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/rangerkms_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.196.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.196.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:46:01 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613961, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:46:01.043913 31504 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:46:01.055778 31504 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:46:02.083774 31504 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:46:02.083873 31504 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms
I20260928 16:46:02.083990 31504 mini_ranger_kms.cc:227] Using postgres at 127.30.196.20:36123
2026-09-28 16:46:02,624 [I] DB FLAVOR :POSTGRES
2026-09-28 16:46:02,625 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:46:02,625 [I] Checking connection
2026-09-28 16:46:02,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:46:02,880 [I] connection success
2026-09-28 16:46:02,880 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:46:02,880 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:46:02,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:46:03,139 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:46:03,139 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:46:03,139 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:46:03,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:36123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:46:03,413 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:46:03.919760 31504 mini_ranger_kms.cc:310] Created kms service
I20260928 16:46:04.138145 31504 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:46:04.188107 31504 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:46:04 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613961, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:04.649978 31504 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:46:04.650126 31504 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms:/tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:46:04.650156 31504 mini_ranger_kms.cc:240] Using host: 127.30.196.20
I20260928 16:46:04.655047 31504 mini_ranger_kms.cc:276] Ranger KMS PID: 10232
I20260928 16:46:04.655200 31504 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.196.20:55089
Sep 28, 2026 4:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:46:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:46:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:46:06.987036 31504 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.943s user 0.027s sys 0.247s
I20260928 16:46:06.987190 31504 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:46:06.987332 31504 mini_ranger_kms.cc:401] 127.30.196.20:55089/kms/v1/keys
Sep 28 16:46:07 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613967, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:46:07 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613967, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
Sep 28 16:46:07 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613967, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
Sep 28 16:46:10 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:46:10.292737 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34303
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34303
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:55089/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:10.428386 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:10.428661 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:10.428702 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:10.432691 10282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:10.432752 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:10.432771 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:10.432787 10282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:10.432802 10282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:10.438258 10282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.196.20:55089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34303
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:10.439333 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:10.440434 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:10.449327 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:10.449314 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:10.449505 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:10.449528 10282 server_base.cc:1034] running on GCE node
I20260928 16:46:10.450325 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:10.451427 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:10.452589 10282 hybrid_clock.cc:648] HybridClock initialized: now 1790613970452564 us; error 39 us; skew 500 ppm
Sep 28 16:46:10 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613970, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:10.461258 10282 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:46:10.462242 10282 webserver.cc:460] Webserver started at http://127.30.196.62:46465/ using document root <none> and password file <none>
I20260928 16:46:10.462839 10282 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:10.462898 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:10.463143 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:10 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613970, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:10.527021 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "981a345a46804e7b8e3e20b0fb7e544d"
format_stamp: "Formatted at 2026-09-28 16:46:10 on dist-test-slave-5rg1"
server_key: "65f4eaec01d3b06e767023482904162f"
server_key_iv: "af086fd168454341bc2a881ad2ad1ece"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:10.527838 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "981a345a46804e7b8e3e20b0fb7e544d"
format_stamp: "Formatted at 2026-09-28 16:46:10 on dist-test-slave-5rg1"
server_key: "65f4eaec01d3b06e767023482904162f"
server_key_iv: "af086fd168454341bc2a881ad2ad1ece"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:10.533293 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:46:10.551682 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:10.553320 10282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:10.553463 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "981a345a46804e7b8e3e20b0fb7e544d"
format_stamp: "Formatted at 2026-09-28 16:46:10 on dist-test-slave-5rg1"
server_key: "65f4eaec01d3b06e767023482904162f"
server_key_iv: "af086fd168454341bc2a881ad2ad1ece"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:10.553620 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:10.574587 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:10.578228 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:10.578438 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:10.587090 10282 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34303
I20260928 16:46:10.587160 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34303 every 8 connection(s)
I20260928 16:46:10.588331 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:10.591225 10350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:10.592619 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 10282
I20260928 16:46:10.592757 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:46:10.593066 31504 external_mini_cluster.cc:1428] Setting key 4fdec0c62bf99a445c5a0962032e3c05
I20260928 16:46:10.598168 10350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: Bootstrap starting.
I20260928 16:46:10.600715 10350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:10.601563 10350 log.cc:824] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:46:10 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:46:10.603999 10350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: No bootstrap required, opened a new log
I20260928 16:46:10.608129 10350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:46:10.608512 10350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:10.608649 10350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981a345a46804e7b8e3e20b0fb7e544d, State: Initialized, Role: FOLLOWER
I20260928 16:46:10.609213 10350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:46:10.609372 10350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:10.609458 10350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:10.609566 10350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:10.609661 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:10.594663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.30.196.62:34303)
0928 16:46:10.595192 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:10.595212 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:10.595258 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:46:10.596172 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:10.596197 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:10.596635 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:10.597000 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:10.597912 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:10.599375 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:10.600579 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:10.601252 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:10.601407 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:10.604241 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:10.604247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:10.604260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:10.604287 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:10.606580 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:10.607032 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:10.607038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:10.607050 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:10.607155 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:10.607440 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:10.607445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:10.607446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:10.607926 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:10.608140 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:10.608401 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 16:46:10.608719 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":156,"threads_started":1}
I20260928 16:46:10.610719 10350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:46:10.611101 10350 leader_election.cc:302] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981a345a46804e7b8e3e20b0fb7e544d; no voters:
I20260928 16:46:10.611436 10350 leader_election.cc:288] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:46:10.612718 10350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:10.615147 10350 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:46:10.615913 10356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:10.616166 10356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [term 1 LEADER]: Becoming Leader. State: Replica: 981a345a46804e7b8e3e20b0fb7e544d, State: Running, Role: LEADER
I20260928 16:46:10.616587 10356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } }
I20260928 16:46:10.620574 10367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981a345a46804e7b8e3e20b0fb7e544d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } } }
I20260928 16:46:10.620882 10367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:10.621151 10367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981a345a46804e7b8e3e20b0fb7e544d. Latest consensus state: current_term: 1 leader_uuid: "981a345a46804e7b8e3e20b0fb7e544d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981a345a46804e7b8e3e20b0fb7e544d" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34303 } } }
I20260928 16:46:10.621236 10367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:10.625952 10371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:10.629921 10371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:10.641891 10371 catalog_manager.cc:1347] Generated new cluster ID: 1569e01ea4b74dae9ffa9b638a9771af
I20260928 16:46:10.642073 10371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:10.654503 10371 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:10.656021 10371 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:10.665802 10371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: Generated new TSK 0
I20260928 16:46:10.666791 10371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:46:11.387701 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:55089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:11.525686 10395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:11.526137 10395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:11.526257 10395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:11.531147 10395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:11.531301 10395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:11.531452 10395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:46:11.539480 10395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.196.20:55089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:11.541157 10395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:11.542464 10395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:11.554739 10403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:11.554752 10405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:11.554806 10402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:11.555316 10395 server_base.cc:1034] running on GCE node
I20260928 16:46:11.555886 10395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:11.556694 10395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:11.557902 10395 hybrid_clock.cc:648] HybridClock initialized: now 1790613971557862 us; error 48 us; skew 500 ppm
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:11.561375 10395 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:46:11.562476 10395 webserver.cc:460] Webserver started at http://127.30.196.1:38467/ using document root <none> and password file <none>
I20260928 16:46:11.563098 10395 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:11.563197 10395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:11.563465 10395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:11.587916 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "8c4ab41857fa430b8e7c8bb133e0628e"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "b7a9424b4328a63b39b9a3c32f01ee0d"
server_key_iv: "bee7ad3c6282204d510a07e3d0088080"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.588776 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "8c4ab41857fa430b8e7c8bb133e0628e"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "b7a9424b4328a63b39b9a3c32f01ee0d"
server_key_iv: "bee7ad3c6282204d510a07e3d0088080"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.594252 10395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:46:11.617695 10412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:11.619603 10395 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.002s
I20260928 16:46:11.619793 10395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "8c4ab41857fa430b8e7c8bb133e0628e"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "b7a9424b4328a63b39b9a3c32f01ee0d"
server_key_iv: "bee7ad3c6282204d510a07e3d0088080"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.619959 10395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:11.667068 10395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:11.671042 10395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:11.671322 10395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:11.672124 10395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:11.673250 10395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:11.673357 10395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:11.673482 10395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:11.673560 10395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:11.685148 10395 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:36101
I20260928 16:46:11.685284 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:36101 every 8 connection(s)
I20260928 16:46:11.686225 10395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:11.687925 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 10395
I20260928 16:46:11.688069 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:46:11.688329 31504 external_mini_cluster.cc:1428] Setting key 9d83686169028c11139389e9052bc427
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:46:11.701753 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:11.690318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:43503 (local address 127.30.196.62:34303)
0928 16:46:11.690600 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:11.690606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:11.690622 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:46:11.691035 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:11.691038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:11.691094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:11.691185 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:11.692633 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:11.693539 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:11.694779 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:11.695021 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:11.695135 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:11.697648 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:11.697652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:11.697654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:11.697682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:11.699294 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:11.700011 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:11.700015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:11.700017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:11.700104 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:11.700475 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:11.700479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:11.700481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:11.700681 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:11.700792 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:11.701387 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 16:46:11.701545 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260928 16:46:11.702705 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:11.689196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.1:43503)
0928 16:46:11.689761 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:11.689808 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:11.690736 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:11.691351 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:11.691357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:11.691740 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:11.692451 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:11.692474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:11.693672 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:11.693676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:11.694648 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:11.694658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:11.695177 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:11.695183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:11.695369 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:11.696038 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:11.696055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:11.697453 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:11.699470 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:11.699476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:11.699497 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:11.699883 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:11.700200 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:11.700205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:11.700207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:11.700370 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:11.700791 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:11.700801 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:11.701128 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 16:46:11.701333 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 16:46:11.701701 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":167,"threads_started":1}
I20260928 16:46:11.704324 10529 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
I20260928 16:46:11.704632 10529 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:11.705240 10529 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
I20260928 16:46:11.707170 10314 ts_manager.cc:194] Registered new tserver with Master: 8c4ab41857fa430b8e7c8bb133e0628e (127.30.196.1:36101)
I20260928 16:46:11.708889 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:43503
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:46:11.735448 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:55089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:46:11.878257 10535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:11.878567 10535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:11.878688 10535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:11.883152 10535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:11.883246 10535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:11.883352 10535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:46:11.889375 10535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.196.20:55089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:11.890667 10535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:11.891939 10535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:11.904350 10541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:11.904605 10544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:11.904866 10542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:11.905314 10535 server_base.cc:1034] running on GCE node
I20260928 16:46:11.905748 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:11.906505 10535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:11.909842 10535 hybrid_clock.cc:648] HybridClock initialized: now 1790613971909816 us; error 44 us; skew 500 ppm
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:11.912993 10535 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:46:11.914187 10535 webserver.cc:460] Webserver started at http://127.30.196.2:35875/ using document root <none> and password file <none>
I20260928 16:46:11.914997 10535 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:11.915155 10535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:11.915457 10535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:11 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:11.940164 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "2fb125879427a93d2fa2530fe80cc84b"
server_key_iv: "42cb729af5c48cbd0aad9ba49caf1638"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.940976 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "2fb125879427a93d2fa2530fe80cc84b"
server_key_iv: "42cb729af5c48cbd0aad9ba49caf1638"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.947645 10535 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:46:11.983417 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:11.985313 10535 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260928 16:46:11.985478 10535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92"
format_stamp: "Formatted at 2026-09-28 16:46:11 on dist-test-slave-5rg1"
server_key: "2fb125879427a93d2fa2530fe80cc84b"
server_key_iv: "42cb729af5c48cbd0aad9ba49caf1638"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:11.985632 10535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:12.010442 10535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:12.014837 10535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:12.015082 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:12.015799 10535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:12.016839 10535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:12.016944 10535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:12.017041 10535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:12.017104 10535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:12.031769 10535 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:40585
I20260928 16:46:12.033159 10535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:12.034679 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 10535
I20260928 16:46:12.034828 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:46:12.035187 31504 external_mini_cluster.cc:1428] Setting key 059b0fadbe0d831705887925c226e261
I20260928 16:46:12.034709 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:40585 every 8 connection(s)
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:46:12.058650 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.039850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:56951 (local address 127.30.196.62:34303)
0928 16:46:12.039975 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.039980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.039997 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:46:12.040740 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.040744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.040804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.040866 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:12.042529 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.043346 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.045128 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.045498 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.045599 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.054084 (+ 8485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:12.054089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:12.054092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:12.054126 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:12.055947 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.056654 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.056658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.056660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.056749 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.057202 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.057206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.057207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.057465 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:12.057570 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.058304 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.058450 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:46:12.060741 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.038676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.2:56951)
0928 16:46:12.039284 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.039349 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.040442 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.041284 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.041292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.041658 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:12.042338 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.042358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.043523 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.043527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.044976 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.044987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.047598 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.047602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.047863 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.051839 (+ 3976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:12.051863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:12.053822 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:12.056105 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:12.056112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:12.056132 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:12.056523 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:12.056860 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:12.056864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:12.056866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:12.057074 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:12.057644 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:12.057651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:12.058006 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.059175 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.059569 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":154,"threads_started":1}
I20260928 16:46:12.062359 10666 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
I20260928 16:46:12.062626 10666 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:12.063226 10666 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
I20260928 16:46:12.064484 10314 ts_manager.cc:194] Registered new tserver with Master: 8ba7d406fa1c4ee3a1720dc9936e0d92 (127.30.196.2:40585)
I20260928 16:46:12.065254 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:56951
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
W20260928 16:46:12.137606 31504 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.30.196.3: real 0.100s user 0.000s sys 0.004s
I20260928 16:46:12.138314 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:55089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
W20260928 16:46:12.425899 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:12.426426 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:12.426558 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:12.433516 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:12.433686 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:12.433832 10671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:46:12.443353 10671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.196.20:55089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:12.445300 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:12.446873 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:12.460827 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:12.462348 10671 server_base.cc:1034] running on GCE node
W20260928 16:46:12.462827 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:12.463852 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:12.464505 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:12.465421 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:12.472863 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790613972472764 us; error 108 us; skew 500 ppm
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613972, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:12.476919 10671 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:46:12.478166 10671 webserver.cc:460] Webserver started at http://127.30.196.3:38855/ using document root <none> and password file <none>
I20260928 16:46:12.478982 10671 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:12.479398 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:12.479769 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613972, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:12.497439 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "55c925656973447a8a9e5c8ed57bb71e"
format_stamp: "Formatted at 2026-09-28 16:46:12 on dist-test-slave-5rg1"
server_key: "d16f1ec6ae411b3097fc1cf3af4ccbb2"
server_key_iv: "d2382ad55748c345a70ab979a4b512d6"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:12.498600 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "55c925656973447a8a9e5c8ed57bb71e"
format_stamp: "Formatted at 2026-09-28 16:46:12 on dist-test-slave-5rg1"
server_key: "d16f1ec6ae411b3097fc1cf3af4ccbb2"
server_key_iv: "d2382ad55748c345a70ab979a4b512d6"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:12.523074 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.004s sys 0.000s
I20260928 16:46:12.545841 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:12.547590 10671 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 16:46:12.547760 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "55c925656973447a8a9e5c8ed57bb71e"
format_stamp: "Formatted at 2026-09-28 16:46:12 on dist-test-slave-5rg1"
server_key: "d16f1ec6ae411b3097fc1cf3af4ccbb2"
server_key_iv: "d2382ad55748c345a70ab979a4b512d6"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:12.547919 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:12.585101 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:12.595749 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:12.596063 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:12.596794 10671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:12.597863 10671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:12.597965 10671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:12.598049 10671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:12.598095 10671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:12.614212 10671 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:45239
I20260928 16:46:12.615545 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:12.617447 10800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:45239 every 8 connection(s)
I20260928 16:46:12.624567 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 10671
I20260928 16:46:12.624706 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:46:12.625003 31504 external_mini_cluster.cc:1428] Setting key fb4534ec846b311abdd636d98566e198
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613972, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:46:12.657323 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.620146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34303 (local address 127.30.196.3:45083)
0928 16:46:12.624026 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.624094 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.625295 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.630563 (+ 5268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.630574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.631012 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:12.631788 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.631822 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.633021 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.633024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.634399 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.634411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.635164 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.635168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.635387 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.648889 (+ 13502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:12.648925 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:12.650780 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:12.653297 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:12.653306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:12.653328 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:12.653746 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:12.654119 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:12.654122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:12.654124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:12.654515 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:12.655091 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:12.655099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:12.655469 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.655764 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.656172 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3608,"thread_start_us":181,"threads_started":1}
I20260928 16:46:12.658118 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.621302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:45083 (local address 127.30.196.62:34303)
0928 16:46:12.630059 (+ 8757us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.630063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.630076 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:46:12.630116 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.630118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.630167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.631963 (+ 1796us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:12.632067 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.632847 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.634545 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.634880 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.634984 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.651049 (+ 16065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:12.651053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:12.651056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:12.651090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:12.653020 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.653884 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.653889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.653891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.654001 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.654640 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.654644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.654646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.654926 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:12.655030 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.657781 (+ 2751us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.657949 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8661,"thread_start_us":83,"threads_started":1}
I20260928 16:46:12.659756 10801 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34303
I20260928 16:46:12.660014 10801 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:12.660601 10801 heartbeater.cc:511] Master 127.30.196.62:34303 requested a full tablet report, sending...
I20260928 16:46:12.661923 10314 ts_manager.cc:194] Registered new tserver with Master: 55c925656973447a8a9e5c8ed57bb71e (127.30.196.3:45239)
I20260928 16:46:12.662665 10314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:45083
I20260928 16:46:12.665939 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:46:12.697309 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.676195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.30.196.62:34303)
0928 16:46:12.676332 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.676337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.688053 (+ 11716us) server_negotiation.cc:407] Connection header received
0928 16:46:12.688269 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.688273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.688326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.688378 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:12.689413 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.690202 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.691397 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.691761 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.691838 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.693672 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:12.693677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:12.693679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:12.693708 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:12.695370 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.695874 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.695878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.695880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.695961 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:12.696257 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:12.696261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:12.696262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:12.696469 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:12.696573 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.697015 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.697149 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:46:12.701546 10314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:46:12.706579 10314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:46:12.712461 10529 heartbeater.cc:503] Master 127.30.196.62:34303 was elected leader, sending a full tablet report...
I20260928 16:46:12.766264 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.752545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:45239 (local address 127.0.0.1:33886)
0928 16:46:12.755345 (+ 2800us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.755358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.755455 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.756833 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.756839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.756867 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:12.757025 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.757034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.759251 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.759254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.761937 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.761946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.765685 (+ 3739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.765689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.765820 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.765871 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.765982 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.766109 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2742,"thread_start_us":108,"threads_started":1}
I20260928 16:46:12.766884 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.752966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.30.196.3:45239)
0928 16:46:12.755030 (+ 2064us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.755042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.755569 (+ 527us) server_negotiation.cc:407] Connection header received
0928 16:46:12.755653 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.755658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.755900 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.756059 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:12.757161 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.758494 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.762103 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.763020 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.763128 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.763274 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.766629 (+ 3355us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.766742 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":132,"threads_started":1}
I20260928 16:46:12.770543 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.753821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:40585 (local address 127.0.0.1:46846)
0928 16:46:12.761456 (+ 7635us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.761471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.761560 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.762251 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.762254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.762271 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:12.762364 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.762370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.763658 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.763683 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.766307 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.766319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.770121 (+ 3802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.770125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.770233 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.770259 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.770337 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.770409 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7570,"mutex_wait_us":1500,"thread_start_us":91,"threads_started":1}
I20260928 16:46:12.770192 10734 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 (DEFAULT_TABLE table=test-table [id=c13c73f0ae7b41a1b9e1182593c3eeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:12.771394 10734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8ba325c72448dbee9f1ba7796f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:12.771852 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.754031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.30.196.2:40585)
0928 16:46:12.755208 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.755213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.761662 (+ 6449us) server_negotiation.cc:407] Connection header received
0928 16:46:12.761749 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.761754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.761951 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.762107 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:12.762490 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.763536 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.766451 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.767371 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.767470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.770051 (+ 2581us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.770774 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.771710 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":91,"threads_started":1}
I20260928 16:46:12.774719 10598 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 (DEFAULT_TABLE table=test-table [id=c13c73f0ae7b41a1b9e1182593c3eeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:12.775883 10598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8ba325c72448dbee9f1ba7796f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:12.792250 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.751341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:36101 (local address 127.0.0.1:33004)
0928 16:46:12.754542 (+ 3201us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.754597 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.754731 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.758655 (+ 3924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.758658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.758679 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:12.758776 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.758782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.763349 (+ 4567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.763353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.764600 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.764610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.791620 (+ 27010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.791625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.791873 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.791911 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.791981 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.792080 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2370,"mutex_wait_us":1,"thread_start_us":145,"threads_started":1}
I20260928 16:46:12.793119 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.751483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.30.196.1:36101)
0928 16:46:12.756384 (+ 4901us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.756390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.756413 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:46:12.756501 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.756506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.756681 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.759004 (+ 2323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:12.759736 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.760838 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.764757 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.765573 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.792606 (+ 27033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.792798 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.792860 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.792973 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4790,"thread_start_us":127,"threads_started":1}
I20260928 16:46:12.795073 10820 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Bootstrap starting.
I20260928 16:46:12.796473 10462 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 (DEFAULT_TABLE table=test-table [id=c13c73f0ae7b41a1b9e1182593c3eeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:12.797686 10820 tablet_bootstrap.cc:654] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:12.797734 10462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8ba325c72448dbee9f1ba7796f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:12.798568 10820 log.cc:824] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:12.799499 10821 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Bootstrap starting.
I20260928 16:46:12.802686 10821 tablet_bootstrap.cc:654] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:12.803601 10821 log.cc:824] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:12.806113 10820 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: No bootstrap required, opened a new log
I20260928 16:46:12.806346 10820 ts_tablet_manager.cc:1397] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260928 16:46:12.811152 10820 raft_consensus.cc:348] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.811457 10820 raft_consensus.cc:374] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:12.811533 10820 raft_consensus.cc:729] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba7d406fa1c4ee3a1720dc9936e0d92, State: Initialized, Role: FOLLOWER
I20260928 16:46:12.814039 10821 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: No bootstrap required, opened a new log
I20260928 16:46:12.814342 10821 ts_tablet_manager.cc:1397] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260928 16:46:12.818580 10821 raft_consensus.cc:348] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.819259 10820 consensus_queue.cc:260] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.820637 10820 ts_tablet_manager.cc:1428] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260928 16:46:12.821075 10821 raft_consensus.cc:374] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:12.821118 10666 heartbeater.cc:503] Master 127.30.196.62:34303 was elected leader, sending a full tablet report...
I20260928 16:46:12.821199 10821 raft_consensus.cc:729] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c925656973447a8a9e5c8ed57bb71e, State: Initialized, Role: FOLLOWER
I20260928 16:46:12.821714 10821 consensus_queue.cc:260] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.822721 10821 ts_tablet_manager.cc:1428] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260928 16:46:12.823442 10801 heartbeater.cc:503] Master 127.30.196.62:34303 was elected leader, sending a full tablet report...
I20260928 16:46:12.824807 10824 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Bootstrap starting.
I20260928 16:46:12.827517 10824 tablet_bootstrap.cc:654] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:12.828598 10824 log.cc:824] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:12.831759 10824 tablet_bootstrap.cc:492] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: No bootstrap required, opened a new log
I20260928 16:46:12.832073 10824 ts_tablet_manager.cc:1397] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260928 16:46:12.836287 10824 raft_consensus.cc:348] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.836711 10824 raft_consensus.cc:374] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:12.836802 10824 raft_consensus.cc:729] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4ab41857fa430b8e7c8bb133e0628e, State: Initialized, Role: FOLLOWER
I20260928 16:46:12.837404 10824 consensus_queue.cc:260] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.838616 10824 ts_tablet_manager.cc:1428] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 16:46:12.868189 10802 tablet.cc:2369] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:46:12 dist-test-slave-5rg1 krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613971, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
W20260928 16:46:12.938081 10530 tablet.cc:2369] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:12.953599 10825 raft_consensus.cc:482] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:46:12.953871 10825 raft_consensus.cc:504] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.955341 10825 leader_election.cc:288] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c4ab41857fa430b8e7c8bb133e0628e (127.30.196.1:36101), 55c925656973447a8a9e5c8ed57bb71e (127.30.196.3:45239)
I20260928 16:46:12.964541 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.956667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:57579 (local address 127.30.196.1:36101)
0928 16:46:12.956800 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.956806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.960013 (+ 3207us) server_negotiation.cc:407] Connection header received
0928 16:46:12.960183 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.960186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.960240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.960321 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:12.961335 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.961928 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.963171 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.963758 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.963862 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.963925 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.964282 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.964366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:46:12.965214 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.955759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:36101 (local address 127.30.196.2:57579)
0928 16:46:12.959897 (+ 4138us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.959916 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.960073 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.960942 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.960945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.960983 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:12.961152 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.961159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.962089 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.962092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.962976 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.963033 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.964031 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.964034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.964137 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.964160 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.964967 (+ 807us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.965052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4044,"thread_start_us":138,"threads_started":1}
I20260928 16:46:12.966075 10482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8ba325c72448dbee9f1ba7796f8f6" candidate_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" is_pre_election: true
I20260928 16:46:12.966418 10482 raft_consensus.cc:2454] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba7d406fa1c4ee3a1720dc9936e0d92 in term 0.
I20260928 16:46:12.967088 10554 leader_election.cc:302] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba7d406fa1c4ee3a1720dc9936e0d92, 8c4ab41857fa430b8e7c8bb133e0628e; no voters:
I20260928 16:46:12.967429 10825 raft_consensus.cc:2781] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:46:12.967530 10825 raft_consensus.cc:482] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:46:12.967582 10825 raft_consensus.cc:3037] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:12.968034 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.956921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:36699 (local address 127.30.196.3:45239)
0928 16:46:12.957043 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.957047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.963804 (+ 6757us) server_negotiation.cc:407] Connection header received
0928 16:46:12.964025 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.964028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.964085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.964136 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:12.964539 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.965260 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.966658 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.967234 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.967333 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.967384 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:12.967790 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 16:46:12.967891 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:46:12.968381 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.956206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:45239 (local address 127.30.196.2:36699)
0928 16:46:12.963703 (+ 7497us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:12.963718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:12.963881 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:12.964280 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:12.964283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:12.964301 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:12.964400 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.964406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.965429 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.965435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.966507 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:12.966518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:12.967483 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:12.967486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:12.967595 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:12.967622 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:46:12.968162 (+ 540us) client_negotiation.cc:240] Negotiation successful
0928 16:46:12.968243 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7396,"thread_start_us":85,"threads_started":1}
I20260928 16:46:12.969200 10825 raft_consensus.cc:504] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.969528 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8ba325c72448dbee9f1ba7796f8f6" candidate_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c925656973447a8a9e5c8ed57bb71e" is_pre_election: true
I20260928 16:46:12.969609 10825 leader_election.cc:288] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [CANDIDATE]: Term 1 election: Requested vote from peers 8c4ab41857fa430b8e7c8bb133e0628e (127.30.196.1:36101), 55c925656973447a8a9e5c8ed57bb71e (127.30.196.3:45239)
I20260928 16:46:12.969843 10748 raft_consensus.cc:2454] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba7d406fa1c4ee3a1720dc9936e0d92 in term 0.
I20260928 16:46:12.970093 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8ba325c72448dbee9f1ba7796f8f6" candidate_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c925656973447a8a9e5c8ed57bb71e"
I20260928 16:46:12.970227 10753 raft_consensus.cc:3037] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:12.970939 10482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8ba325c72448dbee9f1ba7796f8f6" candidate_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4ab41857fa430b8e7c8bb133e0628e"
I20260928 16:46:12.971088 10482 raft_consensus.cc:3037] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:12.971798 10753 raft_consensus.cc:2454] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba7d406fa1c4ee3a1720dc9936e0d92 in term 1.
I20260928 16:46:12.972481 10552 leader_election.cc:302] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c925656973447a8a9e5c8ed57bb71e, 8ba7d406fa1c4ee3a1720dc9936e0d92; no voters:
I20260928 16:46:12.972630 10482 raft_consensus.cc:2454] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba7d406fa1c4ee3a1720dc9936e0d92 in term 1.
I20260928 16:46:12.972672 10825 raft_consensus.cc:2781] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:12.972898 10825 raft_consensus.cc:686] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba7d406fa1c4ee3a1720dc9936e0d92, State: Running, Role: LEADER
I20260928 16:46:12.973250 10825 consensus_queue.cc:237] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } }
I20260928 16:46:12.976788 10314 catalog_manager.cc:5495] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba7d406fa1c4ee3a1720dc9936e0d92 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba7d406fa1c4ee3a1720dc9936e0d92" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40585 } health_report { overall_health: HEALTHY } } }
I20260928 16:46:13.004768 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:12.999228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.30.196.2:40585)
0928 16:46:12.999445 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:12.999450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:12.999468 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:46:12.999572 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:12.999575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:12.999650 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:12.999804 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:46:13.000227 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:13.001034 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:13.002378 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:13.002711 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:13.002823 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:13.003294 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:46:13.003939 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:46:13.004067 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:13.004549 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 16:46:13.004618 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 16:46:13.013578 10753 raft_consensus.cc:1261] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 1 FOLLOWER]: Refusing update from remote peer 8ba7d406fa1c4ee3a1720dc9936e0d92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:13.014659 10833 consensus_queue.cc:1035] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c925656973447a8a9e5c8ed57bb71e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:13.022507 10482 raft_consensus.cc:1261] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8c4ab41857fa430b8e7c8bb133e0628e [term 1 FOLLOWER]: Refusing update from remote peer 8ba7d406fa1c4ee3a1720dc9936e0d92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:13.023443 10833 consensus_queue.cc:1035] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c4ab41857fa430b8e7c8bb133e0628e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:13.039592 10836 mvcc.cc:205] Tried to move back new op lower bound from 7334354833453182976 to 7334354833301749760. Current Snapshot: MvccSnapshot[applied={T|T < 7334354833453182976 or (T in {7334354833453182976})}]
I20260928 16:46:13.048761 10314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:13.048995 10314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:13.057433 10314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 981a345a46804e7b8e3e20b0fb7e544d: Sending DeleteTablet for 3 replicas of tablet 26f8ba325c72448dbee9f1ba7796f8f6
I20260928 16:46:13.058530 10734 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:13 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:33886
I20260928 16:46:13.058951 10598 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:13 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:46846
I20260928 16:46:13.059199 10462 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8ba325c72448dbee9f1ba7796f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:13 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:33004
I20260928 16:46:13.059731 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 10395
I20260928 16:46:13.059620 10847 tablet_replica.cc:330] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: stopping tablet replica
I20260928 16:46:13.061502 10847 raft_consensus.cc:2229] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:46:13.061908 10847 raft_consensus.cc:2258] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:13.062150 10846 tablet_replica.cc:330] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: stopping tablet replica
I20260928 16:46:13.062716 10846 raft_consensus.cc:2229] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:13.063973 10846 raft_consensus.cc:2258] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:13.064555 10847 ts_tablet_manager.cc:1905] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:13.066118 10846 ts_tablet_manager.cc:1905] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:13.069638 10847 ts_tablet_manager.cc:1918] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:13.075073 10846 ts_tablet_manager.cc:1918] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:13.078985 10847 log.cc:1196] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/26f8ba325c72448dbee9f1ba7796f8f6
I20260928 16:46:13.079560 10847 ts_tablet_manager.cc:1939] T 26f8ba325c72448dbee9f1ba7796f8f6 P 8ba7d406fa1c4ee3a1720dc9936e0d92: Deleting consensus metadata
I20260928 16:46:13.079811 10846 log.cc:1196] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/26f8ba325c72448dbee9f1ba7796f8f6
I20260928 16:46:13.080277 10846 ts_tablet_manager.cc:1939] T 26f8ba325c72448dbee9f1ba7796f8f6 P 55c925656973447a8a9e5c8ed57bb71e: Deleting consensus metadata
I20260928 16:46:13.081904 10299 catalog_manager.cc:4841] TS 8ba7d406fa1c4ee3a1720dc9936e0d92 (127.30.196.2:40585): tablet 26f8ba325c72448dbee9f1ba7796f8f6 (table test-table [id=c13c73f0ae7b41a1b9e1182593c3eeb6]) successfully deleted
I20260928 16:46:13.083508 10298 catalog_manager.cc:4841] TS 55c925656973447a8a9e5c8ed57bb71e (127.30.196.3:45239): tablet 26f8ba325c72448dbee9f1ba7796f8f6 (table test-table [id=c13c73f0ae7b41a1b9e1182593c3eeb6]) successfully deleted
W20260928 16:46:13.083901 10301 connection.cc:535] server connection from 127.30.196.1:43503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 16:46:13.084501 10300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:36101 (error 108)
I20260928 16:46:13.085664 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 10535
I20260928 16:46:13.089129 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:13.088692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:36101 (local address 127.0.0.1:33010)
0928 16:46:13.088934 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:13.089017 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:36101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 16:46:13.089380 10300 catalog_manager.cc:4584] TS 8c4ab41857fa430b8e7c8bb133e0628e (127.30.196.1:36101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26f8ba325c72448dbee9f1ba7796f8f6: Network error: Client connection negotiation failed: client connection to 127.30.196.1:36101: connect: Connection refused (error 111)
I20260928 16:46:13.094056 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 10671
I20260928 16:46:13.104730 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 10282
I20260928 16:46:13.111500 31504 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:46:13.456960 31504 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:46:13.457098 31504 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:46:13.802603 31504 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:46:13.802 UTC [9868] LOG: received smart shutdown request
2026-09-28 16:46:13.806 UTC [9868] LOG: background worker "logical replication launcher" (PID 9877) exited with exit code 1
2026-09-28 16:46:13.807 UTC [9872] LOG: shutting down
2026-09-28 16:46:13.989 UTC [9868] LOG: database system is shut down
2026-09-28T16:46:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49272 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:46:14 dist-test-slave-5rg1 krb5kdc[10858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:46:14 dist-test-slave-5rg1 krb5kdc[10858](info): set up 2 sockets
Sep 28 16:46:14 dist-test-slave-5rg1 krb5kdc[10858](info): commencing operation
krb5kdc: starting...
W20260928 16:46:15.180852 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:46:15 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:46:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangeradmin_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangeradmin_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangerlookup_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangerlookup_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
I20260928 16:46:15.358093 31504 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6WsIs2/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/postgres -l logfile start
2026-09-28 16:46:18.089 UTC [10888] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:46:18.089 UTC [10888] LOG: listening on IPv4 address "127.30.196.20", port 56071
2026-09-28 16:46:18.093 UTC [10888] LOG: listening on Unix socket "/tmp/.s.PGSQL.56071"
2026-09-28 16:46:18.116 UTC [10891] LOG: database system was shut down at 2026-09-28 16:46:16 UTC
2026-09-28 16:46:18.121 UTC [10888] LOG: database system is ready to accept connections
I20260928 16:46:19.088326 31504 mini_postgres.cc:96] Postgres bound to 56071
2026-09-28 16:46:19.094 UTC [10899] FATAL: database "slave" does not exist
127.30.196.20:56071 - accepting connections
I20260928 16:46:19.095427 31504 mini_ranger.cc:161] Starting Ranger...
I20260928 16:46:19.111092 31504 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:46:19.657683 31504 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:46:19.657775 31504 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin
2026-09-28 16:46:19,956 [I] DB FLAVOR :POSTGRES
2026-09-28 16:46:19,956 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:46:19,957 [I] Checking connection..
2026-09-28 16:46:19,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:46:20,225 [I] Checking connection passed.
2026-09-28 16:46:20,226 [I] --------- Verifying version history table ---------
2026-09-28 16:46:20,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:46:20,510 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:46:20,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:46:20,780 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:46:20,780 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:46:20,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:46:21.062 UTC [10963] WARNING: there is no transaction in progress
2026-09-28 16:46:21,069 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:46:21,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:46:21,338 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:46:21,339 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:46:21,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:46:21,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:46:21,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5rg1.c.gcp-upstream.internal','N') ;"
2026-09-28 16:46:22,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:46:22,351 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:46:22,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:46:22,603 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:46:22,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:46:22,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:46:23,098 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:46:23,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:46:24.146 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.165 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.184 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.200 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.217 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.236 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.302 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.309 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.325 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.336 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.347 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.362 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.373 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.381 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.389 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:24.397 UTC [11071] WARNING: there is no transaction in progress
2026-09-28 16:46:25,141 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:46:25,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:46:25,400 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:46:25,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:46:25,651 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:46:25,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:46:25,909 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:46:25,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:46:26,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:46:26,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5rg1.c.gcp-upstream.internal';"
2026-09-28 16:46:26,679 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:46:26,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:46:26,931 [I] DB_PATCHES have already been applied
I20260928 16:46:26.937023 31504 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6WsIs2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:46:26.937095 31504 mini_ranger.cc:193] Using host: 127.30.196.20
I20260928 16:46:26.941112 31504 mini_ranger.cc:227] Ranger admin URL: http://127.30.196.20:35797
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:46:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:46:28.093303 31504 mini_ranger.cc:160] Time spent starting Ranger: real 8.998s user 0.000s sys 0.011s
Sep 28, 2026 4:46:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:46:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:46:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:46:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:46:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:46:49.874397 31504 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.196.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.196.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangerkms_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.196.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/rangerkms_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.196.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.196.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.196.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/HTTP_127.30.196.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:46:49 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:46:49.968007 31504 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:46:49.978341 31504 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:46:50.963724 31504 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:46:50.963811 31504 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms
I20260928 16:46:50.963835 31504 mini_ranger_kms.cc:227] Using postgres at 127.30.196.20:56071
2026-09-28 16:46:51,154 [I] DB FLAVOR :POSTGRES
2026-09-28 16:46:51,154 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:46:51,154 [I] Checking connection
2026-09-28 16:46:51,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:46:51,416 [I] connection success
2026-09-28 16:46:51,416 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:46:51,417 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:46:51,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:46:51,674 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:46:51,674 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:46:51,674 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:46:51,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.196.20:56071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:46:51,943 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:46:52.481364 31504 mini_ranger_kms.cc:310] Created kms service
I20260928 16:46:52.710808 31504 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:46:52.760551 31504 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:46:52 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:53.179308 31504 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:46:53.179482 31504 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms:/tmp/dist-test-task6WsIs2/build/debug/bin/postgresql.jar:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6WsIs2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:46:53.179507 31504 mini_ranger_kms.cc:240] Using host: 127.30.196.20
I20260928 16:46:53.183364 31504 mini_ranger_kms.cc:276] Ranger KMS PID: 11251
I20260928 16:46:53.183459 31504 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.196.20:43845
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:46:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 16:46:55 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:55.467746 31504 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.500s user 0.023s sys 0.150s
I20260928 16:46:55.467902 31504 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:46:55.467983 31504 mini_ranger_kms.cc:401] 127.30.196.20:43845/kms/v1/keys
Sep 28 16:46:55 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
Sep 28 16:46:56 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614015, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.196.20@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
Sep 28 16:46:58 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:46:58.452327 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:42785
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:42785
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:58.570663 11302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:58.570955 11302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:58.570994 11302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:58.575121 11302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:58.575193 11302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:58.575213 11302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:58.575229 11302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:58.575244 11302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:58.581044 11302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.196.20:43845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:42785
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:58.582154 11302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:58.583184 11302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:46:58.592796 11302 server_base.cc:1034] running on GCE node
W20260928 16:46:58.592730 11308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:58.592738 11307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:58.593077 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:58.593672 11302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:58.594849 11302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:58.596040 11302 hybrid_clock.cc:648] HybridClock initialized: now 1790614018596022 us; error 37 us; skew 500 ppm
Sep 28 16:46:58 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614018, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:58.615907 11302 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:46:58.617115 11302 webserver.cc:460] Webserver started at http://127.30.196.62:37753/ using document root <none> and password file <none>
I20260928 16:46:58.617715 11302 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:58.617774 11302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:58.618005 11302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:58 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614018, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:58.682490 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "975011ede5ba4c49958663bf691162e3"
format_stamp: "Formatted at 2026-09-28 16:46:58 on dist-test-slave-5rg1"
server_key: "e52000571ae710841f3ddaa84d4658c4"
server_key_iv: "64c22966223ebcda4a60f09c09f6e4aa"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:58.683127 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "975011ede5ba4c49958663bf691162e3"
format_stamp: "Formatted at 2026-09-28 16:46:58 on dist-test-slave-5rg1"
server_key: "e52000571ae710841f3ddaa84d4658c4"
server_key_iv: "64c22966223ebcda4a60f09c09f6e4aa"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:58.687291 11302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:46:58.702459 11317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:58.703963 11302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:58.704123 11302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "975011ede5ba4c49958663bf691162e3"
format_stamp: "Formatted at 2026-09-28 16:46:58 on dist-test-slave-5rg1"
server_key: "e52000571ae710841f3ddaa84d4658c4"
server_key_iv: "64c22966223ebcda4a60f09c09f6e4aa"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:58.704263 11302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:58.733110 11302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:58.736369 11302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:58.736591 11302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:58.745059 11302 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:42785
I20260928 16:46:58.745088 11369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:42785 every 8 connection(s)
I20260928 16:46:58.746052 11302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:58.749415 11370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:58.751097 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11302
I20260928 16:46:58.751230 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:46:58.751531 31504 external_mini_cluster.cc:1428] Setting key cf0a2a7d30cd3aae3517f082676c72ee
I20260928 16:46:58.756845 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Bootstrap starting.
I20260928 16:46:58.759701 11370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Neither blocks nor log segments found. Creating new log.
Sep 28 16:46:58 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:46:58.760643 11370 log.cc:824] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:58.763320 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: No bootstrap required, opened a new log
I20260928 16:46:58.767218 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:58.753103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.30.196.62:42785)
0928 16:46:58.753518 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:58.753539 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:58.753574 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:46:58.754220 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:58.754237 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:58.754572 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:58.754904 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:58.755763 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:58.756756 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:58.757729 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:58.758201 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:58.758371 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:58.761070 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:58.761080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:58.761100 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:58.761136 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:58.763970 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:58.764512 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:58.764518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:58.764525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:58.764651 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:58.764937 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:58.764940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:58.764942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:58.765474 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:58.765685 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:58.765944 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:46:58.766234 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260928 16:46:58.768275 11370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:46:58.768554 11370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:58.768638 11370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975011ede5ba4c49958663bf691162e3, State: Initialized, Role: FOLLOWER
I20260928 16:46:58.769126 11370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:46:58.769527 11370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:58.769589 11370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:58.769706 11370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:58.770649 11370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:46:58.771029 11370 leader_election.cc:302] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975011ede5ba4c49958663bf691162e3; no voters:
I20260928 16:46:58.771438 11370 leader_election.cc:288] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:46:58.771528 11375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:58.771778 11375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 LEADER]: Becoming Leader. State: Replica: 975011ede5ba4c49958663bf691162e3, State: Running, Role: LEADER
I20260928 16:46:58.772150 11375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:46:58.772748 11370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:58.773808 11376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "975011ede5ba4c49958663bf691162e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } } }
I20260928 16:46:58.773932 11376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:58.774137 11376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975011ede5ba4c49958663bf691162e3. Latest consensus state: current_term: 1 leader_uuid: "975011ede5ba4c49958663bf691162e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } } }
I20260928 16:46:58.774201 11376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:58.774998 11384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:58.775117 11370 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:46:58.780942 11384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:58.791507 11384 catalog_manager.cc:1347] Generated new cluster ID: 18005646273640d3a835e2b0f25be4e5
I20260928 16:46:58.791581 11384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:58.812614 11384 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:58.814106 11384 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:58.828725 11384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Generated new TSK 0
I20260928 16:46:58.829769 11384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:46:59.631649 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:59.786181 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:59.786576 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:59.786653 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:59.793103 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:59.793195 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:59.793298 11415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:46:59.801535 11415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:46:59.803062 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:59.804486 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:59.817657 11422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:59.817816 11423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:59.817675 11425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:59.818177 11415 server_base.cc:1034] running on GCE node
I20260928 16:46:59.818735 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:59.819595 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:59.821000 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790614019820868 us; error 83 us; skew 500 ppm
Sep 28 16:46:59 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614019, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:59.824653 11415 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:46:59.825750 11415 webserver.cc:460] Webserver started at http://127.30.196.1:35533/ using document root <none> and password file <none>
I20260928 16:46:59.826524 11415 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:59.826625 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:59.826917 11415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:59 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614019, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:46:59.844523 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "81c3a701684a4e2db849edc1dea12a0c"
format_stamp: "Formatted at 2026-09-28 16:46:59 on dist-test-slave-5rg1"
server_key: "821115325673514a47ad9fba72c3d2bb"
server_key_iv: "576d7b635893054311a8bdb55e8e8334"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:59.845786 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "81c3a701684a4e2db849edc1dea12a0c"
format_stamp: "Formatted at 2026-09-28 16:46:59 on dist-test-slave-5rg1"
server_key: "821115325673514a47ad9fba72c3d2bb"
server_key_iv: "576d7b635893054311a8bdb55e8e8334"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:59.857438 11415 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260928 16:46:59.891067 11434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:59.896854 11415 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.001s sys 0.000s
I20260928 16:46:59.897048 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "81c3a701684a4e2db849edc1dea12a0c"
format_stamp: "Formatted at 2026-09-28 16:46:59 on dist-test-slave-5rg1"
server_key: "821115325673514a47ad9fba72c3d2bb"
server_key_iv: "576d7b635893054311a8bdb55e8e8334"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:59.897207 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:59.927726 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:59.931599 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:59.931923 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:59.932647 11415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:59.933832 11415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:59.933997 11415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:59.934134 11415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:59.934216 11415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:59.948405 11415 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:46185
I20260928 16:46:59.949787 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:59.953217 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11415
I20260928 16:46:59.953352 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:46:59.953701 31504 external_mini_cluster.cc:1428] Setting key a83b3f187c597b606d87b59058e9f891
I20260928 16:46:59.951293 11547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:46185 every 8 connection(s)
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
Sep 28 16:46:59 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614019, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:46:59 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614019, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:46:59.987067 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:59.953903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.1:43693)
0928 16:46:59.964022 (+ 10119us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:59.964108 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:59.965647 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:59.966780 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:59.966791 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:59.967207 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:59.967979 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:59.968000 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:59.969138 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:59.969142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:59.970715 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:59.970731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:59.971797 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:59.971801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:59.972050 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:59.973040 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:59.973068 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:59.978716 (+ 5648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:59.983244 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:59.983254 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:59.983277 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:59.983813 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:59.984187 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:59.984190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:59.984193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:59.984425 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:59.984934 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:59.984943 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:59.985311 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 16:46:59.985534 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 16:46:59.985919 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9772,"thread_start_us":135,"threads_started":1}
I20260928 16:46:59.988245 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:59.953421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:43693 (local address 127.30.196.62:42785)
0928 16:46:59.966194 (+ 12773us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:59.966199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:59.966216 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:46:59.966264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:59.966267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:59.966322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:59.966415 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:59.968171 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:59.968977 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:59.970891 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:59.971433 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:59.971559 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:59.979009 (+ 7450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:59.979013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:59.979017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:59.979054 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:59.983038 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:59.983958 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:59.983963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:59.983965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:59.984076 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:59.984538 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:59.984542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:59.984544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:59.984785 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:59.984874 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:59.987907 (+ 3033us) server_negotiation.cc:299] Negotiation successful
0928 16:46:59.988065 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12672,"thread_start_us":98,"threads_started":1}
I20260928 16:46:59.990407 11548 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:46:59.990685 11548 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:59.991276 11548 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:46:59.993085 11334 ts_manager.cc:194] Registered new tserver with Master: 81c3a701684a4e2db849edc1dea12a0c (127.30.196.1:46185)
I20260928 16:46:59.994612 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:43693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:47:00.016271 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:00.181757 11554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:00.182271 11554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:00.182404 11554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:00.188836 11554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:00.189023 11554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:00.189175 11554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:00.197654 11554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:00.199448 11554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:00.200978 11554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:00.215399 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.215989 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.216835 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:00.217519 11554 server_base.cc:1034] running on GCE node
I20260928 16:47:00.218152 11554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:00.218981 11554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:00.220229 11554 hybrid_clock.cc:648] HybridClock initialized: now 1790614020220152 us; error 89 us; skew 500 ppm
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:00.224232 11554 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:00.225477 11554 webserver.cc:460] Webserver started at http://127.30.196.2:35199/ using document root <none> and password file <none>
I20260928 16:47:00.226315 11554 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:00.226477 11554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:00.226800 11554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:00.249610 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "5e5bc09d34f14e568f119741bea08e8a"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "fad53d82775d100c9c07c5ed6635d361"
server_key_iv: "09ac8e877df645f0245fc9877303b96b"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.251967 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "5e5bc09d34f14e568f119741bea08e8a"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "fad53d82775d100c9c07c5ed6635d361"
server_key_iv: "09ac8e877df645f0245fc9877303b96b"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.257771 11554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:47:00.278457 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.280337 11554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:00.280565 11554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "5e5bc09d34f14e568f119741bea08e8a"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "fad53d82775d100c9c07c5ed6635d361"
server_key_iv: "09ac8e877df645f0245fc9877303b96b"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.280804 11554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:00.314869 11554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:00.318887 11554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:00.319164 11554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:00.319931 11554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:00.321012 11554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:00.321128 11554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.321224 11554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:00.321285 11554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.331487 11554 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:33605
I20260928 16:47:00.332804 11554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:00.337756 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11554
I20260928 16:47:00.337895 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:47:00.338198 31504 external_mini_cluster.cc:1428] Setting key d0ff17a85d773a26b62defc74c1ff94b
I20260928 16:47:00.343899 11683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:33605 every 8 connection(s)
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:47:00.368245 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.346369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33013 (local address 127.30.196.62:42785)
0928 16:47:00.346514 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:00.346519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:00.347743 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 16:47:00.350453 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:00.350457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:00.350516 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:00.350568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:00.352226 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.353245 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.355126 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.355515 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.355598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:00.360946 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:00.360951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:00.360954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:00.360991 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:00.363017 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:00.366154 (+ 3137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:00.366159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:00.366161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:00.366280 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:00.366770 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:00.366774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:00.366776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:00.367038 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:00.367126 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:00.367891 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 16:47:00.368051 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:47:00.370230 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.346948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.2:33013)
0928 16:47:00.347552 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:00.347613 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:00.350162 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:00.350987 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:00.350996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:00.351363 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:00.352049 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:00.352068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.353450 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.353454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:00.354965 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:00.354977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.355854 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.355858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:00.356088 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:00.356860 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:00.356883 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:00.360662 (+ 3779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:00.363185 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:00.363192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:00.363214 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:00.363790 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:00.366415 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:00.366419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:00.366421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:00.366656 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:00.367196 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:00.367203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:00.367581 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 16:47:00.368753 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0928 16:47:00.369151 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":164,"threads_started":1}
I20260928 16:47:00.371840 11684 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:47:00.372087 11684 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:00.372709 11684 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:00.373963 11334 ts_manager.cc:194] Registered new tserver with Master: 5e5bc09d34f14e568f119741bea08e8a (127.30.196.2:33605)
I20260928 16:47:00.374717 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:33013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:47:00.404224 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614019, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
W20260928 16:47:00.595256 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:00.595639 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:00.595721 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:00.601676 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:00.601763 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:00.601847 11689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:00.609835 11689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:00.611438 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:00.612835 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:00.625170 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.625170 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.625420 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:00.625684 11689 server_base.cc:1034] running on GCE node
I20260928 16:47:00.626180 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:00.626993 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:00.628197 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790614020628171 us; error 38 us; skew 500 ppm
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:00.631894 11689 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:00.635064 11689 webserver.cc:460] Webserver started at http://127.30.196.3:39801/ using document root <none> and password file <none>
I20260928 16:47:00.635864 11689 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:00.635958 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:00.636194 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:00.661767 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "a8ec9bd0ecb64f728fba5cd08c69125e"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "9fdb94bd9b5b780b7e3f46d41da8d0e3"
server_key_iv: "c8ec06077dbb06eb2b0ba32f1cfb1215"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.662643 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "a8ec9bd0ecb64f728fba5cd08c69125e"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "9fdb94bd9b5b780b7e3f46d41da8d0e3"
server_key_iv: "c8ec06077dbb06eb2b0ba32f1cfb1215"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.668497 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:47:00.686443 11704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.688323 11689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:47:00.688479 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "a8ec9bd0ecb64f728fba5cd08c69125e"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "9fdb94bd9b5b780b7e3f46d41da8d0e3"
server_key_iv: "c8ec06077dbb06eb2b0ba32f1cfb1215"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.688620 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:00.708488 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:00.712577 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:00.712837 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:00.713610 11689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:00.714711 11689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:00.714833 11689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.714942 11689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:00.715023 11689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.726858 11689 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:46863
I20260928 16:47:00.726963 11817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:46863 every 8 connection(s)
I20260928 16:47:00.728258 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:00.731149 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11689
I20260928 16:47:00.731271 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:47:00.731602 31504 external_mini_cluster.cc:1428] Setting key b5f1be97b171522154156cfe3782fac9
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:00.749549 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.731909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:60525 (local address 127.30.196.62:42785)
0928 16:47:00.732063 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:00.732069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:00.736271 (+ 4202us) server_negotiation.cc:407] Connection header received
0928 16:47:00.737634 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:00.737637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:00.737693 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:00.737779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:00.739349 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.740177 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.741898 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.742243 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.742345 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:00.745515 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:00.745519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:00.745522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:00.745550 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:00.747236 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:00.747867 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:00.747870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:00.747872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:00.747953 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:00.748324 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:00.748330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:00.748332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:00.748549 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:00.748623 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:00.749268 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 16:47:00.749396 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:47:00.751435 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.732666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.3:60525)
0928 16:47:00.736076 (+ 3410us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:00.736144 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:00.737367 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:00.737974 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:00.737984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:00.738423 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:00.739168 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:00.739192 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.740320 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.740324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:00.741769 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:00.741781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:00.742418 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:00.742422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:00.742641 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:00.743364 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:00.743395 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:00.745311 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:00.747359 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:00.747365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:00.747392 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:00.747759 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:00.748058 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:00.748061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:00.748063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:00.748225 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:00.748687 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:00.748694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:00.749012 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 16:47:00.750082 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0928 16:47:00.750422 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3144,"thread_start_us":194,"threads_started":1}
I20260928 16:47:00.752972 11818 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:47:00.753211 11818 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:00.753698 11818 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:00.754707 11334 ts_manager.cc:194] Registered new tserver with Master: a8ec9bd0ecb64f728fba5cd08c69125e (127.30.196.3:46863)
I20260928 16:47:00.755260 11334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:60525
I20260928 16:47:00.757803 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:00.758514 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11415
I20260928 16:47:00.765522 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11554
I20260928 16:47:00.771447 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11689
I20260928 16:47:00.777290 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11302
I20260928 16:47:00.789439 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:42785
--webserver_interface=127.30.196.62
--webserver_port=37753
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:42785
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:00.901818 11821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:00.902127 11821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:00.902168 11821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:00.906350 11821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:00.906418 11821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:00.906440 11821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:00.906456 11821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:00.906493 11821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:00.912286 11821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:42785
--ranger_config_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=37753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:00.913371 11821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:00.914438 11821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:00.923278 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.923278 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:00.923305 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:00.923732 11821 server_base.cc:1034] running on GCE node
I20260928 16:47:00.924381 11821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:00.925730 11821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:00.926949 11821 hybrid_clock.cc:648] HybridClock initialized: now 1790614020926929 us; error 41 us; skew 500 ppm
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:00.930043 11821 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:00.930980 11821 webserver.cc:460] Webserver started at http://127.30.196.62:37753/ using document root <none> and password file <none>
I20260928 16:47:00.931528 11821 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:00.931598 11821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:47:00 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614020, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:00.987810 11821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:47:00.990428 11837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:00.991963 11821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:00.992126 11821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "975011ede5ba4c49958663bf691162e3"
format_stamp: "Formatted at 2026-09-28 16:46:58 on dist-test-slave-5rg1"
server_key: "e52000571ae710841f3ddaa84d4658c4"
server_key_iv: "64c22966223ebcda4a60f09c09f6e4aa"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:00.992755 11821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:01.014367 11821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:01.017699 11821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:01.018213 11821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:01.025674 11821 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:42785
I20260928 16:47:01.025681 11889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:42785 every 8 connection(s)
I20260928 16:47:01.026747 11821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:01.027249 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11821
I20260928 16:47:01.027710 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:46185
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:42785
--webserver_port=35533
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:01.031456 11890 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:01.033237 11890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Bootstrap starting.
I20260928 16:47:01.050557 11890 log.cc:824] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:01.057171 11890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:01.057597 11890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Bootstrap complete.
I20260928 16:47:01.061738 11890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:47:01.062125 11890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975011ede5ba4c49958663bf691162e3, State: Initialized, Role: FOLLOWER
I20260928 16:47:01.062616 11890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:47:01.062728 11890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:01.062806 11890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:01.062901 11890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:47:01.064854 11890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:47:01.065177 11890 leader_election.cc:302] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975011ede5ba4c49958663bf691162e3; no voters:
I20260928 16:47:01.065465 11890 leader_election.cc:288] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:47:01.065605 11894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:47:01.065908 11894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [term 2 LEADER]: Becoming Leader. State: Replica: 975011ede5ba4c49958663bf691162e3, State: Running, Role: LEADER
I20260928 16:47:01.066282 11894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } }
I20260928 16:47:01.066608 11890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:01.067812 11896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975011ede5ba4c49958663bf691162e3. Latest consensus state: current_term: 2 leader_uuid: "975011ede5ba4c49958663bf691162e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } } }
I20260928 16:47:01.067817 11895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "975011ede5ba4c49958663bf691162e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975011ede5ba4c49958663bf691162e3" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 42785 } } }
I20260928 16:47:01.067938 11896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:01.067953 11895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:01.068243 11903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:01.071589 11903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:01.072168 11890 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:47:01.072568 11903 catalog_manager.cc:1259] Loaded cluster ID: 18005646273640d3a835e2b0f25be4e5
I20260928 16:47:01.072615 11903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:01.074941 11903 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:01.075600 11903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Loaded TSK: 0
I20260928 16:47:01.076853 11903 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:47:01.151266 11892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:01.151590 11892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:01.151662 11892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:01.156450 11892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:01.156531 11892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:01.156630 11892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:01.164155 11892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:46185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=35533
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:01.165422 11892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:01.166513 11892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:01.177915 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:01.178277 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:01.178304 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:01.178866 11892 server_base.cc:1034] running on GCE node
I20260928 16:47:01.179381 11892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:01.180238 11892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:01.181461 11892 hybrid_clock.cc:648] HybridClock initialized: now 1790614021181427 us; error 56 us; skew 500 ppm
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:01.184598 11892 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:01.185714 11892 webserver.cc:460] Webserver started at http://127.30.196.1:35533/ using document root <none> and password file <none>
I20260928 16:47:01.186504 11892 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:01.186612 11892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:01.218647 11892 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:01.228014 11938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.229727 11892 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260928 16:47:01.229892 11892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "81c3a701684a4e2db849edc1dea12a0c"
format_stamp: "Formatted at 2026-09-28 16:46:59 on dist-test-slave-5rg1"
server_key: "821115325673514a47ad9fba72c3d2bb"
server_key_iv: "576d7b635893054311a8bdb55e8e8334"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:01.230556 11892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:01.267879 11892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:01.272310 11892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:01.272555 11892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:01.273285 11892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:01.274281 11892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:01.274360 11892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.274434 11892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:01.274463 11892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.288887 11892 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:46185
I20260928 16:47:01.290376 11892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:01.291411 12051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:46185 every 8 connection(s)
I20260928 16:47:01.299508 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 11892
I20260928 16:47:01.300117 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:33605
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:42785
--webserver_port=35199
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:01.318055 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.294191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:45159 (local address 127.30.196.62:42785)
0928 16:47:01.294712 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:01.294731 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:01.298422 (+ 3691us) server_negotiation.cc:407] Connection header received
0928 16:47:01.300164 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:01.300181 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:01.300625 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:01.301904 (+ 1279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:01.302943 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.304318 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.306165 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.307027 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.307207 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.310911 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:01.310923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:01.310948 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:01.310992 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:01.313616 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.314494 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.314504 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.314513 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.314686 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.315162 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.315166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.315167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.315722 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:01.315834 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:01.316637 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 16:47:01.316971 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260928 16:47:01.327744 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.294330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.1:45159)
0928 16:47:01.298167 (+ 3837us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.298240 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:01.299216 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:01.301191 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:01.301222 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:01.301743 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:01.302688 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.302735 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.304491 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.304497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.306000 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.306016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.307545 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.307552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.307828 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.308591 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:01.308618 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:01.310597 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:01.313801 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.313812 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.313834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.314308 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.314828 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.314834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.314838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.315049 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.315938 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:01.315945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:01.316247 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 16:47:01.326117 (+ 9870us) client_negotiation.cc:240] Negotiation successful
0928 16:47:01.326577 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3586,"thread_start_us":158,"threads_started":1}
I20260928 16:47:01.330863 12052 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:47:01.331136 12052 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.331800 12052 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.334156 12052 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:42785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:47:01.334283 12052 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.334466 12052 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:01.335325 12052 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.335505 12052 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.336270 12052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:47:01.469100 12056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:01.469537 12056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:01.469632 12056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:01.477644 12056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:01.477763 12056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:01.477830 12056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:01.483495 12056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:33605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=35199
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:01.484678 12056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:01.485739 12056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:01.496654 12064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:01.496843 12056 server_base.cc:1034] running on GCE node
W20260928 16:47:01.496647 12061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:01.497133 12062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:01.497546 12056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:01.498349 12056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:01.499542 12056 hybrid_clock.cc:648] HybridClock initialized: now 1790614021499509 us; error 47 us; skew 500 ppm
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:01.503091 12056 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:01.504189 12056 webserver.cc:460] Webserver started at http://127.30.196.2:35199/ using document root <none> and password file <none>
I20260928 16:47:01.504920 12056 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:01.505046 12056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:01.532187 12056 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:01.535796 12072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.537635 12056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:47:01.537889 12056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "5e5bc09d34f14e568f119741bea08e8a"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "fad53d82775d100c9c07c5ed6635d361"
server_key_iv: "09ac8e877df645f0245fc9877303b96b"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:01.538944 12056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:01.561012 12056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:01.565402 12056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:01.565713 12056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:01.566615 12056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:01.567971 12056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:01.568085 12056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.568197 12056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:01.568270 12056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.580271 12056 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:33605
I20260928 16:47:01.580576 12185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:33605 every 8 connection(s)
I20260928 16:47:01.581897 12056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:01.591146 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12056
I20260928 16:47:01.591708 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.196.20:43845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:46863
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:42785
--webserver_port=39801
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:01.604466 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.584080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:32961 (local address 127.30.196.62:42785)
0928 16:47:01.584241 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:01.584246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:01.585374 (+ 1128us) server_negotiation.cc:407] Connection header received
0928 16:47:01.586771 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:01.586776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:01.586845 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:01.586946 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:01.588596 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.589486 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.591485 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.591905 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.592000 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.596373 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:01.596377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:01.596380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:01.596413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:01.598251 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.602050 (+ 3799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.602055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.602057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.602172 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.602811 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.602816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.602819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.603166 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:01.603268 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:01.604128 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 16:47:01.604285 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:47:01.606562 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.584627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.2:32961)
0928 16:47:01.585192 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.585250 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:01.586497 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:01.587164 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:01.587174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:01.587564 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:01.588394 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.588418 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.589653 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.589657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.591214 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.591333 (+ 119us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.592913 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.592918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.593147 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.593865 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:01.593889 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:01.596132 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:01.601365 (+ 5233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.601373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.601395 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.601880 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.602318 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.602321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.602323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.602650 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.603360 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:01.603367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:01.603779 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 16:47:01.604985 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0928 16:47:01.605403 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260928 16:47:01.608295 12186 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:47:01.608564 12186 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.609216 12186 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.611101 12186 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:42785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:47:01.611232 12186 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.611414 12186 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:01.612361 12186 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.612526 12186 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.613302 12186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:47:01.733121 12189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:01.733466 12189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:01.733561 12189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:01.738842 12189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:01.738950 12189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:01.739058 12189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:01.748014 12189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.196.20:43845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=39801
--tserver_master_addrs=127.30.196.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:01.749509 12189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:01.751031 12189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:01.764537 12198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:01.764572 12195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:01.765074 12196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:01.767361 12189 server_base.cc:1034] running on GCE node
I20260928 16:47:01.768132 12189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:01.769032 12189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:01.772315 12189 hybrid_clock.cc:648] HybridClock initialized: now 1790614021772262 us; error 56 us; skew 500 ppm
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:01.776641 12189 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:01.777910 12189 webserver.cc:460] Webserver started at http://127.30.196.3:39801/ using document root <none> and password file <none>
I20260928 16:47:01.778702 12189 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:01.778820 12189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:01.803151 12189 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:47:01.806398 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.808198 12189 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:47:01.808441 12189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "a8ec9bd0ecb64f728fba5cd08c69125e"
format_stamp: "Formatted at 2026-09-28 16:47:00 on dist-test-slave-5rg1"
server_key: "9fdb94bd9b5b780b7e3f46d41da8d0e3"
server_key_iv: "c8ec06077dbb06eb2b0ba32f1cfb1215"
server_key_version: "kuduclusterkey@0"
I20260928 16:47:01.809227 12189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:01.840579 12189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:01.844228 12189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:01.844448 12189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:01.845178 12189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:01.846266 12189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:01.846336 12189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.846441 12189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:01.846484 12189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:01.860417 12189 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:46863
I20260928 16:47:01.861907 12189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:01.862553 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12189
I20260928 16:47:01.866846 12319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:46863 every 8 connection(s)
I20260928 16:47:01.873798 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.863761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.30.196.62:42785)
0928 16:47:01.863896 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:01.863901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:01.864098 (+ 197us) server_negotiation.cc:407] Connection header received
0928 16:47:01.864340 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:01.864344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:01.864401 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:01.864465 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:01.865650 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.866456 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.867750 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.868095 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.868225 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.869944 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:01.869948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:01.869951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:01.869977 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:01.871780 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.872294 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.872297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.872301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.872393 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.872725 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.872728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.872730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.872933 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:01.873127 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:01.873342 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 16:47:01.873474 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:47:01 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614021, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:01.880748 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.866210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:37809 (local address 127.30.196.62:42785)
0928 16:47:01.866457 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:01.866468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:01.869038 (+ 2570us) server_negotiation.cc:407] Connection header received
0928 16:47:01.869083 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:01.869086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:01.869138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:01.869231 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:01.871091 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.871965 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.873471 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.873798 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.873908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.876736 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:01.876739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:01.876741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:01.876763 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:01.878287 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.878936 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.878938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.878940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.879068 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:01.879490 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:01.879493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:01.879496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:01.879732 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:01.879825 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:01.880493 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 16:47:01.880608 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260928 16:47:01.882004 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.865085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:42785 (local address 127.30.196.3:37809)
0928 16:47:01.867564 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.867607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:01.868706 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:01.869556 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:01.869565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:01.869920 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:01.870646 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.870670 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.872087 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.872090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.873338 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:01.873349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:01.873970 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:01.873975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:01.874221 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:01.874808 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:01.874830 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:01.876487 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:01.878421 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.878428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.878446 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.878823 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.879171 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:01.879174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:01.879177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:01.879374 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:01.879844 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:01.879853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:01.880204 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 16:47:01.880452 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 16:47:01.880821 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2272,"thread_start_us":141,"threads_started":1}
I20260928 16:47:01.883432 12320 heartbeater.cc:347] Connected to a master server at 127.30.196.62:42785
I20260928 16:47:01.883752 12320 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.884411 12320 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.886271 12320 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:42785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:47:01.886457 12320 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.886665 12320 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:01.887609 12320 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:01.887830 12320 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
W20260928 16:47:01.888774 12320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 16:47:02 dist-test-slave-5rg1 krb5kdc[10858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790614022, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:02.336508 12052 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:02.336833 12052 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:02.338007 11854 ts_manager.cc:194] Registered new tserver with Master: 81c3a701684a4e2db849edc1dea12a0c (127.30.196.1:46185)
I20260928 16:47:02.339731 11854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:45159
I20260928 16:47:02.613541 12186 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:02.613838 12186 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:02.614641 11854 ts_manager.cc:194] Registered new tserver with Master: 5e5bc09d34f14e568f119741bea08e8a (127.30.196.2:33605)
I20260928 16:47:02.615463 11854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:32961
Sep 28 16:47:02 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614022, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:02.889053 12320 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:02.889389 12320 heartbeater.cc:511] Master 127.30.196.62:42785 requested a full tablet report, sending...
I20260928 16:47:02.890991 11854 ts_manager.cc:194] Registered new tserver with Master: a8ec9bd0ecb64f728fba5cd08c69125e (127.30.196.3:46863)
I20260928 16:47:02.891819 11854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:37809
I20260928 16:47:02.898584 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:47:02 dist-test-slave-5rg1 krb5kdc[10858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614022, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for HTTP/127.30.196.20@KRBTEST.COM
I20260928 16:47:02.938450 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.902582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.30.196.62:42785)
0928 16:47:02.907945 (+ 5363us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:02.907950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:02.907972 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:02.908033 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:02.908039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:02.908101 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:02.909223 (+ 1122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:02.909350 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.912066 (+ 2716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.913042 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.927079 (+ 14037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.927251 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:02.929519 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:02.929522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:02.929524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:02.929559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:02.934405 (+ 4846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:02.935306 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:02.935309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:02.935311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:02.935436 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:02.936693 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:02.936698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:02.936703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:02.937160 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:02.938038 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:02.938121 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 16:47:02.938257 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5248,"thread_start_us":93,"threads_started":1}
I20260928 16:47:02.943162 11854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:02.947291 11854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:02.984803 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.977543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:33605 (local address 127.0.0.1:39274)
0928 16:47:02.978480 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.978496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:02.978591 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:02.979550 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:02.979552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:02.979568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:02.979643 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:02.979647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.981002 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.981005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:02.982864 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:02.982870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.984186 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.984192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:02.984314 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:02.984358 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:47:02.984482 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:47:02.984625 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260928 16:47:02.984994 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.975994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:46863 (local address 127.0.0.1:40506)
0928 16:47:02.976806 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.976861 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:02.977012 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:02.978859 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:02.978865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:02.978903 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:02.979064 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:02.979072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.981541 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.981545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:02.982675 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:02.982688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.984611 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.984614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:02.984703 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:02.984727 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:47:02.984774 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 16:47:02.984840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260928 16:47:02.985553 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.975987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.30.196.3:46863)
0928 16:47:02.978349 (+ 2362us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:02.978357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:02.978383 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:47:02.978461 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:02.978467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:02.978685 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:02.979246 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:02.979398 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.981412 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.982828 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.984477 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.985107 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:02.985253 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:02.985309 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 16:47:02.985412 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2239,"thread_start_us":131,"threads_started":1}
I20260928 16:47:02.986876 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.977980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.30.196.2:33605)
0928 16:47:02.978849 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:02.978856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:02.978879 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:47:02.978968 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:02.978973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:02.979167 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:02.979322 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:02.979823 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.980840 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.982999 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:02.983826 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:02.983949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:02.984112 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:02.986630 (+ 2518us) server_negotiation.cc:299] Negotiation successful
0928 16:47:02.986736 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":70,"threads_started":1}
I20260928 16:47:02.989265 12254 tablet_service.cc:1467] Processing CreateTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb (DEFAULT_TABLE table=test-table [id=73bab593f3dd4c848dacf598a22ccb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:02.990491 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb (DEFAULT_TABLE table=test-table [id=73bab593f3dd4c848dacf598a22ccb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:02.991534 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da1994a74c84701ad8a7cd0baa810eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:02.991539 12254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da1994a74c84701ad8a7cd0baa810eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:03.001493 12356 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Bootstrap starting.
I20260928 16:47:03.002228 12355 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Bootstrap starting.
I20260928 16:47:03.004294 12356 tablet_bootstrap.cc:654] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:03.005301 12355 tablet_bootstrap.cc:654] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:03.005366 12356 log.cc:824] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:03.006166 12355 log.cc:824] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:03.007786 12356 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: No bootstrap required, opened a new log
I20260928 16:47:03.007997 12356 ts_tablet_manager.cc:1397] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:47:03.015556 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.976827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:46185 (local address 127.0.0.1:34984)
0928 16:47:02.977264 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.977276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:02.977349 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:02.986601 (+ 9252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:02.986605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:02.986627 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:02.986740 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:02.986747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.000478 (+ 13731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.000483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.001682 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:03.001696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.014817 (+ 13121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.014822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.015151 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.015195 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:47:03.015281 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:47:03.015380 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":115,"threads_started":1}
I20260928 16:47:03.017818 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.976973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.30.196.1:46185)
0928 16:47:02.979920 (+ 2947us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:02.979927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:02.979949 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:02.980022 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:02.980027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:02.986429 (+ 6402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:02.989312 (+ 2883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:02.989501 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.000296 (+ 10795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.013499 (+ 13203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.014612 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.017191 (+ 2579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.017409 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:03.017480 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 16:47:03.017613 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2828,"thread_start_us":81,"threads_started":1}
I20260928 16:47:03.017913 12355 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: No bootstrap required, opened a new log
I20260928 16:47:03.018498 12356 raft_consensus.cc:348] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.021246 12356 raft_consensus.cc:374] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:03.021377 12356 raft_consensus.cc:729] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5bc09d34f14e568f119741bea08e8a, State: Initialized, Role: FOLLOWER
I20260928 16:47:03.021327 11985 tablet_service.cc:1467] Processing CreateTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb (DEFAULT_TABLE table=test-table [id=73bab593f3dd4c848dacf598a22ccb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:03.021945 12356 consensus_queue.cc:260] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.022565 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da1994a74c84701ad8a7cd0baa810eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:03.023010 12356 ts_tablet_manager.cc:1428] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260928 16:47:03.023371 12186 heartbeater.cc:503] Master 127.30.196.62:42785 was elected leader, sending a full tablet report...
I20260928 16:47:03.023818 12355 ts_tablet_manager.cc:1397] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.002s
I20260928 16:47:03.030382 12355 raft_consensus.cc:348] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.030714 12355 raft_consensus.cc:374] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:03.030789 12355 raft_consensus.cc:729] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ec9bd0ecb64f728fba5cd08c69125e, State: Initialized, Role: FOLLOWER
I20260928 16:47:03.031275 12355 consensus_queue.cc:260] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.032349 12355 ts_tablet_manager.cc:1428] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260928 16:47:03.032719 12320 heartbeater.cc:503] Master 127.30.196.62:42785 was elected leader, sending a full tablet report...
I20260928 16:47:03.047149 12361 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: Bootstrap starting.
I20260928 16:47:03.054280 12361 tablet_bootstrap.cc:654] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:03.055399 12361 log.cc:824] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:03.058434 12361 tablet_bootstrap.cc:492] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: No bootstrap required, opened a new log
I20260928 16:47:03.058665 12361 ts_tablet_manager.cc:1397] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260928 16:47:03.062961 12361 raft_consensus.cc:348] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.063270 12361 raft_consensus.cc:374] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:03.063315 12361 raft_consensus.cc:729] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c3a701684a4e2db849edc1dea12a0c, State: Initialized, Role: FOLLOWER
I20260928 16:47:03.064054 12361 consensus_queue.cc:260] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.065351 12361 ts_tablet_manager.cc:1428] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:47:03.065788 12052 heartbeater.cc:503] Master 127.30.196.62:42785 was elected leader, sending a full tablet report...
W20260928 16:47:03.087481 12187 tablet.cc:2369] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:03.110459 12359 raft_consensus.cc:482] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:03.110754 12359 raft_consensus.cc:504] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.112403 12359 leader_election.cc:288] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ec9bd0ecb64f728fba5cd08c69125e (127.30.196.3:46863), 81c3a701684a4e2db849edc1dea12a0c (127.30.196.1:46185)
W20260928 16:47:03.117734 12321 tablet.cc:2369] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:03.120052 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.113971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:43363 (local address 127.30.196.3:46863)
0928 16:47:03.114288 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:03.114293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:03.114310 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:47:03.114371 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:03.114375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:03.114430 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:03.114538 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:03.114953 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.115979 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.118630 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.119239 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.119346 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.119400 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:03.119803 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 16:47:03.119913 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260928 16:47:03.120707 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.112849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:46863 (local address 127.30.196.2:43363)
0928 16:47:03.113221 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.113239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:03.113360 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:03.114623 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:03.114626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:03.114644 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:03.114795 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:03.114802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.116133 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.116136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.118482 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:03.118494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.119504 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.119507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.119616 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.119640 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:47:03.120450 (+ 810us) client_negotiation.cc:240] Negotiation successful
0928 16:47:03.120544 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260928 16:47:03.121578 12263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da1994a74c84701ad8a7cd0baa810eb" candidate_uuid: "5e5bc09d34f14e568f119741bea08e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" is_pre_election: true
I20260928 16:47:03.121954 12263 raft_consensus.cc:2454] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5bc09d34f14e568f119741bea08e8a in term 0.
I20260928 16:47:03.122618 12074 leader_election.cc:302] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5bc09d34f14e568f119741bea08e8a, a8ec9bd0ecb64f728fba5cd08c69125e; no voters:
I20260928 16:47:03.123238 12359 raft_consensus.cc:2781] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:03.123332 12359 raft_consensus.cc:482] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:03.123389 12359 raft_consensus.cc:3037] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:03.124857 12359 raft_consensus.cc:504] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.125260 12359 leader_election.cc:288] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [CANDIDATE]: Term 1 election: Requested vote from peers a8ec9bd0ecb64f728fba5cd08c69125e (127.30.196.3:46863), 81c3a701684a4e2db849edc1dea12a0c (127.30.196.1:46185)
I20260928 16:47:03.125950 12263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da1994a74c84701ad8a7cd0baa810eb" candidate_uuid: "5e5bc09d34f14e568f119741bea08e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e"
I20260928 16:47:03.126085 12263 raft_consensus.cc:3037] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:03.127585 12263 raft_consensus.cc:2454] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5bc09d34f14e568f119741bea08e8a in term 1.
I20260928 16:47:03.128072 12074 leader_election.cc:302] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5bc09d34f14e568f119741bea08e8a, a8ec9bd0ecb64f728fba5cd08c69125e; no voters:
I20260928 16:47:03.128317 12359 raft_consensus.cc:2781] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:03.128545 12359 raft_consensus.cc:686] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 1 LEADER]: Becoming Leader. State: Replica: 5e5bc09d34f14e568f119741bea08e8a, State: Running, Role: LEADER
I20260928 16:47:03.128961 12359 consensus_queue.cc:237] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } }
I20260928 16:47:03.132679 11853 catalog_manager.cc:5495] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5bc09d34f14e568f119741bea08e8a (127.30.196.2). New cstate: current_term: 1 leader_uuid: "5e5bc09d34f14e568f119741bea08e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5bc09d34f14e568f119741bea08e8a" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 33605 } health_report { overall_health: HEALTHY } } }
I20260928 16:47:03.145138 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.114135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:47615 (local address 127.30.196.1:46185)
0928 16:47:03.114292 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:03.114297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:03.114313 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:47:03.114369 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:03.114372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:03.114419 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:03.114496 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:03.117494 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.118235 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.138320 (+ 20085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.138902 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.139010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.139068 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:03.144859 (+ 5791us) server_negotiation.cc:299] Negotiation successful
0928 16:47:03.144992 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:47:03.145925 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.113342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:46185 (local address 127.30.196.2:47615)
0928 16:47:03.113715 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.113731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:03.113824 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:03.117176 (+ 3352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:03.117180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:03.117208 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:03.117350 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:03.117357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.136670 (+ 19313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.136674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.138110 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:03.138128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.144532 (+ 6404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.144536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:03.144665 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.144698 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:47:03.145651 (+ 953us) client_negotiation.cc:240] Negotiation successful
0928 16:47:03.145751 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260928 16:47:03.146849 12006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da1994a74c84701ad8a7cd0baa810eb" candidate_uuid: "5e5bc09d34f14e568f119741bea08e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c3a701684a4e2db849edc1dea12a0c" is_pre_election: true
I20260928 16:47:03.147239 12006 raft_consensus.cc:2454] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5bc09d34f14e568f119741bea08e8a in term 0.
I20260928 16:47:03.147595 12001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da1994a74c84701ad8a7cd0baa810eb" candidate_uuid: "5e5bc09d34f14e568f119741bea08e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c3a701684a4e2db849edc1dea12a0c"
I20260928 16:47:03.147743 12001 raft_consensus.cc:3037] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:03.149441 12001 raft_consensus.cc:2454] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5bc09d34f14e568f119741bea08e8a in term 1.
I20260928 16:47:03.158473 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.153041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.30.196.2:33605)
0928 16:47:03.153165 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:03.153169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:03.153183 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:47:03.153264 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:03.153267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:03.153332 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:03.153442 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:03.153832 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.154649 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.156147 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:03.156490 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:03.156593 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:03.157043 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:03.157652 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:03.157769 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:03.158235 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 16:47:03.158302 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:47:03.166342 12001 raft_consensus.cc:1261] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 1 FOLLOWER]: Refusing update from remote peer 5e5bc09d34f14e568f119741bea08e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:03.167208 12366 consensus_queue.cc:1035] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c3a701684a4e2db849edc1dea12a0c" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:03.174568 12263 raft_consensus.cc:1261] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 1 FOLLOWER]: Refusing update from remote peer 5e5bc09d34f14e568f119741bea08e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:03.175289 12366 consensus_queue.cc:1035] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ec9bd0ecb64f728fba5cd08c69125e" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:03.188939 12369 mvcc.cc:205] Tried to move back new op lower bound from 7334355038879707136 to 7334355038740054016. Current Snapshot: MvccSnapshot[applied={T|T < 7334355038879707136 or (T in {7334355038879707136})}]
I20260928 16:47:03.191991 12372 mvcc.cc:205] Tried to move back new op lower bound from 7334355038879707136 to 7334355038740054016. Current Snapshot: MvccSnapshot[applied={T|T < 7334355038879707136}]
I20260928 16:47:03.198884 12370 mvcc.cc:205] Tried to move back new op lower bound from 7334355038879707136 to 7334355038740054016. Current Snapshot: MvccSnapshot[applied={T|T < 7334355038879707136}]
I20260928 16:47:03.198834 11853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:03.199038 11853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:03.201949 11853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 975011ede5ba4c49958663bf691162e3: Sending DeleteTablet for 3 replicas of tablet 4da1994a74c84701ad8a7cd0baa810eb
I20260928 16:47:03.202732 11985 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:03 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:34984
I20260928 16:47:03.203356 12378 tablet_replica.cc:330] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c: stopping tablet replica
I20260928 16:47:03.203370 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:03 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:40506
I20260928 16:47:03.203824 12378 raft_consensus.cc:2229] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:03.204221 12378 raft_consensus.cc:2258] T 4da1994a74c84701ad8a7cd0baa810eb P 81c3a701684a4e2db849edc1dea12a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:03.204376 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da1994a74c84701ad8a7cd0baa810eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:03 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39274
I20260928 16:47:03.204855 12380 tablet_replica.cc:330] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a: stopping tablet replica
I20260928 16:47:03.205140 12380 raft_consensus.cc:2229] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:03.205485 12380 raft_consensus.cc:2258] T 4da1994a74c84701ad8a7cd0baa810eb P 5e5bc09d34f14e568f119741bea08e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:03.206265 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11892
I20260928 16:47:03.208667 12379 tablet_replica.cc:330] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: stopping tablet replica
I20260928 16:47:03.209111 12379 raft_consensus.cc:2229] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:03.209559 12379 raft_consensus.cc:2258] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:47:03.215556 11838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:46185 (error 108)
I20260928 16:47:03.216728 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12056
I20260928 16:47:03.218338 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.218019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:46185 (local address 127.0.0.1:34992)
0928 16:47:03.218208 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.218265 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:46185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 16:47:03.218550 11838 catalog_manager.cc:4584] TS 81c3a701684a4e2db849edc1dea12a0c (127.30.196.1:46185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4da1994a74c84701ad8a7cd0baa810eb: Network error: Client connection negotiation failed: client connection to 127.30.196.1:46185: connect: Connection refused (error 111)
I20260928 16:47:03.221130 12379 ts_tablet_manager.cc:1905] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:03.224370 12379 ts_tablet_manager.cc:1918] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:03.224491 12379 log.cc:1196] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/4da1994a74c84701ad8a7cd0baa810eb
I20260928 16:47:03.224859 12379 ts_tablet_manager.cc:1939] T 4da1994a74c84701ad8a7cd0baa810eb P a8ec9bd0ecb64f728fba5cd08c69125e: Deleting consensus metadata
I20260928 16:47:03.226351 11839 catalog_manager.cc:4841] TS a8ec9bd0ecb64f728fba5cd08c69125e (127.30.196.3:46863): tablet 4da1994a74c84701ad8a7cd0baa810eb (table test-table [id=73bab593f3dd4c848dacf598a22ccb25]) successfully deleted
I20260928 16:47:03.230115 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12189
I20260928 16:47:03.233017 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.232705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:33605 (local address 127.0.0.1:39286)
0928 16:47:03.232832 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.232902 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:33605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 16:47:03.238986 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 11821
I20260928 16:47:03.247777 31504 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:47:03.593276 31504 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:47:03.593390 31504 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:47:03.938970 31504 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:47:03.939 UTC [10888] LOG: received smart shutdown request
2026-09-28 16:47:03.944 UTC [10888] LOG: background worker "logical replication launcher" (PID 10897) exited with exit code 1
2026-09-28 16:47:03.944 UTC [10892] LOG: shutting down
2026-09-28 16:47:04.180 UTC [10888] LOG: database system is shut down
2026-09-28T16:47:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50189 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:47:04 dist-test-slave-5rg1 krb5kdc[12390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:47:04 dist-test-slave-5rg1 krb5kdc[12390](info): set up 2 sockets
Sep 28 16:47:04 dist-test-slave-5rg1 krb5kdc[12390](info): commencing operation
krb5kdc: starting...
W20260928 16:47:05.364251 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:47:05.499567 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34681
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:05.612071 12403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:05.612440 12403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:05.612531 12403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:05.616673 12403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:05.616786 12403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:05.616828 12403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:05.616863 12403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:05.616892 12403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:05.622475 12403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:05.623682 12403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:05.624752 12403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:05.633414 12408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:05.633414 12409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:05.633424 12411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:05.634051 12403 server_base.cc:1034] running on GCE node
I20260928 16:47:05.634550 12403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:05.635599 12403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:05.636772 12403 hybrid_clock.cc:648] HybridClock initialized: now 1790614025636741 us; error 42 us; skew 500 ppm
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:05.639572 12403 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:05.640606 12403 webserver.cc:460] Webserver started at http://127.30.196.62:45113/ using document root <none> and password file <none>
I20260928 16:47:05.641199 12403 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:05.641302 12403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:05.641556 12403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:05.643651 12403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "e46c49f4ffda4fc7989cdf1034ec7037"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "5535483897d5cb49244d557c864460b3"
server_key_iv: "145558826fd47657bfd12b13ef641993"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.644335 12403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "e46c49f4ffda4fc7989cdf1034ec7037"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "5535483897d5cb49244d557c864460b3"
server_key_iv: "145558826fd47657bfd12b13ef641993"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.648242 12403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:47:05.650843 12418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:05.652190 12403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:47:05.652377 12403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "e46c49f4ffda4fc7989cdf1034ec7037"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "5535483897d5cb49244d557c864460b3"
server_key_iv: "145558826fd47657bfd12b13ef641993"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.652518 12403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:05.669835 12403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:05.672953 12403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:05.673204 12403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:05.680269 12403 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34681
I20260928 16:47:05.680348 12470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34681 every 8 connection(s)
I20260928 16:47:05.681319 12403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:05.684221 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:05.686534 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12403
I20260928 16:47:05.686640 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:47:05.686941 31504 external_mini_cluster.cc:1428] Setting key 7f1f6212bdffe1630e677f56ac6e4a99
I20260928 16:47:05.688179 12471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.196.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.61@KRBTEST.COM" created.
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:05.704418 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.690448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34681 (local address 127.0.0.1:57692)
0928 16:47:05.690939 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.690991 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:05.691967 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:05.692771 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:05.692776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:05.693097 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:05.693373 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.693385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.695148 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.695152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.695964 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.695974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.696619 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.696622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.696700 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.697271 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:05.697295 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:05.698961 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:05.701307 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.701315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.701323 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.701564 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.701950 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.701952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.701953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.702157 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.702952 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:05.702957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:05.703079 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 16:47:05.703217 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:47:05.703477 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":156,"threads_started":1}
I20260928 16:47:05.704422 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.691342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.30.196.62:34681)
0928 16:47:05.691717 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:05.691730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:05.691779 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:47:05.692291 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:05.692300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:05.692553 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:05.692769 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:05.693538 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.695003 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.696122 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.696481 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.696601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.699129 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:05.699134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:05.699144 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:05.699164 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:05.701199 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.701688 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.701697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.701702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.701848 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.702266 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.702272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.702276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.702829 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:05.702987 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:05.703347 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:47:05.703595 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260928 16:47:05.707299 12471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:47:05.708335 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.708025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45008)
0928 16:47:05.708169 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.708255 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 16:47:05.708571 12420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
I20260928 16:47:05.709523 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.709283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45020)
0928 16:47:05.709412 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.709460 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:47:05.709708 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:33175: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
I20260928 16:47:05.716266 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:33175
--webserver_interface=127.30.196.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:05.759006 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } attempt: 1
I20260928 16:47:05.759934 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.759620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45026)
0928 16:47:05.759786 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.759858 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 16:47:05.760780 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.760560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45036)
0928 16:47:05.760673 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.760726 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:47:05.760915 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:33175: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
I20260928 16:47:05.829106 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } attempt: 2
W20260928 16:47:05.829018 12478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:05.829450 12478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:05.829576 12478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 16:47:05.830014 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.829686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45046)
0928 16:47:05.829854 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.829922 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:47:05.830785 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.830573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45056)
0928 16:47:05.830683 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.830735 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:47:05.830916 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:33175: Network error: Client connection negotiation failed: client connection to 127.30.196.61:33175: connect: Connection refused (error 111)
W20260928 16:47:05.834003 12478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:05.834095 12478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:05.834141 12478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:05.834174 12478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:05.834205 12478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:05.839612 12478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.196.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:05.840842 12478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:05.842133 12478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:05.851740 12483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:05.851727 12484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:05.851715 12486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:05.851789 12478 server_base.cc:1034] running on GCE node
I20260928 16:47:05.852738 12478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:05.853911 12478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:05.855162 12478 hybrid_clock.cc:648] HybridClock initialized: now 1790614025855127 us; error 44 us; skew 500 ppm
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:05.858264 12478 init.cc:377] Logged in from keytab as kudu/127.30.196.61@KRBTEST.COM (short username kudu)
I20260928 16:47:05.859210 12478 webserver.cc:460] Webserver started at http://127.30.196.61:40261/ using document root <none> and password file <none>
I20260928 16:47:05.859862 12478 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:05.859970 12478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:05.860219 12478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:05.862346 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data/instance:
uuid: "b2600256598e4cd8a47643921e39bd2c"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "d5c928ddd19727116573523136bc10af"
server_key_iv: "292892f3ff500711a8fe019555b420ad"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.862941 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal/instance:
uuid: "b2600256598e4cd8a47643921e39bd2c"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "d5c928ddd19727116573523136bc10af"
server_key_iv: "292892f3ff500711a8fe019555b420ad"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.866927 12478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:47:05.869977 12493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:05.871234 12478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:47:05.871409 12478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "b2600256598e4cd8a47643921e39bd2c"
format_stamp: "Formatted at 2026-09-28 16:47:05 on dist-test-slave-5rg1"
server_key: "d5c928ddd19727116573523136bc10af"
server_key_iv: "292892f3ff500711a8fe019555b420ad"
server_key_version: "encryptionkey@0"
I20260928 16:47:05.871591 12478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:05.889873 12478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:05.893380 12478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:05.893615 12478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:05.901265 12478 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.61:33175
I20260928 16:47:05.901279 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.61:33175 every 8 connection(s)
I20260928 16:47:05.902283 12478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
I20260928 16:47:05.903179 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12478
I20260928 16:47:05.903321 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-1/wal/instance
I20260928 16:47:05.903620 31504 external_mini_cluster.cc:1428] Setting key ffe302f7fbbd0d3b4f59781b1c963a85
I20260928 16:47:05.905957 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:05.909957 12546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.196.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.60@KRBTEST.COM" created.
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:05.922098 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } attempt: 3
I20260928 16:47:05.926021 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.912085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.30.196.62:34681)
0928 16:47:05.912248 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:05.912252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:05.913306 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 16:47:05.914283 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:05.914287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:05.914352 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:05.914469 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:05.915723 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.916432 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.917694 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.918165 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.918297 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.921461 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:05.921467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:05.921472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:05.921510 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:05.923697 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.924325 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.924327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.924328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.924404 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.924804 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.924810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.924813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.925001 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:05.925120 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:05.925648 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 16:47:05.925840 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":67}
I20260928 16:47:05.926862 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.912581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34681 (local address 127.0.0.1:57700)
0928 16:47:05.913141 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.913186 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:05.914063 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:05.914660 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:05.914667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:05.915022 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:05.915527 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.915540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.916594 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.916597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.917562 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.917572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.918382 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.918386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.918551 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.919254 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:05.919274 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:05.921245 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:05.923820 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.923828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.923847 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.924220 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.924542 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.924545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.924547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.924695 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.925121 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:05.925129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:05.925384 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 16:47:05.925596 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 16:47:05.925931 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":154,"threads_started":1}
I20260928 16:47:05.928686 12546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:47:05.934026 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.922635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45058)
0928 16:47:05.923046 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.923062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:05.923140 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:05.923872 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:05.923875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:05.924061 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:05.924206 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.924214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.925599 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.925602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.926483 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.926497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.927286 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.927289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.927412 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.927944 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:05.928000 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:05.930213 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:05.932301 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.932304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.932306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.932504 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.932861 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.932864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.932866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.932949 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.933424 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:05.933427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:05.933529 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 16:47:05.933671 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 16:47:05.933814 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
Sep 28 16:47:05 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:47:05.934507 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.922739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.30.196.61:33175)
0928 16:47:05.923414 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:05.923424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:05.923444 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:47:05.923480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:05.923484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:05.923656 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:05.924374 (+ 718us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:05.924617 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.925467 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.926620 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.927170 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.927274 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.930352 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:05.930361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:05.930375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:05.930402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:05.932179 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.932633 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.932642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.932647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.932753 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.933054 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.933058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.933059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.933305 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:05.933459 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:05.933834 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 16:47:05.933955 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"spinlock_wait_cycles":2304,"thread_start_us":229,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
I20260928 16:47:05.935719 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:41753
--webserver_interface=127.30.196.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:05.936589 12471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:47:05.937439 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.937106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50014)
0928 16:47:05.937302 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.937358 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 16:47:05.938091 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.937840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50028)
0928 16:47:05.937992 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.938041 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 16:47:05.938262 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:05.939286 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.929312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.30.196.61:33175)
0928 16:47:05.929664 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:05.929676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:05.929694 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:47:05.929732 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:05.929735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:05.929775 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:05.929867 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:05.930343 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.931111 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.932198 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.932483 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.932606 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.935449 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:05.935456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:05.935462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:05.935503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:05.937433 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.938031 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.938036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.938038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.938117 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:05.938391 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:05.938394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:05.938396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:05.938575 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:05.938654 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:05.938963 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:47:05.939151 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":280,"spinlock_wait_cycles":5376,"thread_start_us":131,"threads_started":1}
I20260928 16:47:05.939379 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.929260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45062)
0928 16:47:05.929398 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.929415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:05.929492 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:05.929944 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:05.929950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:05.930131 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:05.930223 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.930228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.931257 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.931261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.932059 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:05.932069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:05.932603 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:05.932606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:05.932724 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:05.933162 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:05.933175 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:05.934432 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:05.937629 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.937634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.937638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.937901 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.938212 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:05.938217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:05.938222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:05.938310 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:05.938682 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:05.938688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:05.938796 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 16:47:05.938910 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:47:05.939078 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":18}
I20260928 16:47:05.940074 12546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:47:05.940723 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.940500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50044)
0928 16:47:05.940609 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.940674 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:47:05.941011 12497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:05.941880 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.941691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50054)
0928 16:47:05.941797 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.941837 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:47:05.942122 12546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:05.969424 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 1
I20260928 16:47:05.970355 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.970020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50064)
0928 16:47:05.970212 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.970276 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 16:47:05.971127 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.970894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50074)
0928 16:47:05.971019 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.971070 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:47:05.971247 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:05.991362 12546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 1
I20260928 16:47:05.992255 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.991943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50084)
0928 16:47:05.992102 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.992171 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 16:47:05.992972 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.992759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50092)
0928 16:47:05.992873 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.992922 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:47:05.993145 12546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:06.046419 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 2
I20260928 16:47:06.047549 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.047155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50102)
0928 16:47:06.047396 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.047459 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 16:47:06.048364 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.048114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50112)
0928 16:47:06.048257 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.048310 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:47:06.048504 12471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
W20260928 16:47:06.052857 12553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:06.053158 12553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:06.053263 12553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:06.057358 12553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:06.057446 12553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:06.057492 12553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:06.057528 12553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:06.057556 12553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:06.061309 12546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 2
I20260928 16:47:06.062085 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.061796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50126)
0928 16:47:06.061939 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.062005 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:47:06.062774 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.062572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50132)
0928 16:47:06.062675 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.062726 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:47:06.062903 12546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:41753: Network error: Client connection negotiation failed: client connection to 127.30.196.60:41753: connect: Connection refused (error 111)
I20260928 16:47:06.063248 12553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34681,127.30.196.61:33175,127.30.196.60:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.196.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:06.064400 12553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:06.065404 12553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:06.074090 12560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:06.074093 12559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:06.074093 12562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:06.074368 12553 server_base.cc:1034] running on GCE node
I20260928 16:47:06.074970 12553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:06.076210 12553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:06.077407 12553 hybrid_clock.cc:648] HybridClock initialized: now 1790614026077385 us; error 45 us; skew 500 ppm
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614026, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:06.080399 12553 init.cc:377] Logged in from keytab as kudu/127.30.196.60@KRBTEST.COM (short username kudu)
I20260928 16:47:06.081414 12553 webserver.cc:460] Webserver started at http://127.30.196.60:37445/ using document root <none> and password file <none>
I20260928 16:47:06.082008 12553 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:06.082123 12553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:06.082372 12553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:06.084587 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data/instance:
uuid: "eb1fdcfccd1a4c8283e7627929ac00fa"
format_stamp: "Formatted at 2026-09-28 16:47:06 on dist-test-slave-5rg1"
server_key: "12391bd8307e961deec7fbd3a7e02ce7"
server_key_iv: "40d737ef121502ca6c8f11f20ef47647"
server_key_version: "encryptionkey@0"
I20260928 16:47:06.085165 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal/instance:
uuid: "eb1fdcfccd1a4c8283e7627929ac00fa"
format_stamp: "Formatted at 2026-09-28 16:47:06 on dist-test-slave-5rg1"
server_key: "12391bd8307e961deec7fbd3a7e02ce7"
server_key_iv: "40d737ef121502ca6c8f11f20ef47647"
server_key_version: "encryptionkey@0"
I20260928 16:47:06.088919 12553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:06.091351 12569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:06.092648 12553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:06.092800 12553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "eb1fdcfccd1a4c8283e7627929ac00fa"
format_stamp: "Formatted at 2026-09-28 16:47:06 on dist-test-slave-5rg1"
server_key: "12391bd8307e961deec7fbd3a7e02ce7"
server_key_iv: "40d737ef121502ca6c8f11f20ef47647"
server_key_version: "encryptionkey@0"
I20260928 16:47:06.092934 12553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:06.113245 12553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:06.116585 12553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:06.116813 12553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:06.124255 12553 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.60:41753
I20260928 16:47:06.124275 12621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.60:41753 every 8 connection(s)
I20260928 16:47:06.125252 12553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
I20260928 16:47:06.128026 12622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:06.131832 12622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:47:06.133924 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12553
I20260928 16:47:06.134069 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613859608463-31504-0/minicluster-data/master-2/wal/instance
I20260928 16:47:06.134368 31504 external_mini_cluster.cc:1428] Setting key 381331f21a54bc37c4edd1f98dca06cd
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614026, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:47:06.146502 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.133841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.30.196.62:34681)
0928 16:47:06.134041 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.134049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.135182 (+ 1133us) server_negotiation.cc:407] Connection header received
0928 16:47:06.136354 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.136357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.136401 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.136469 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.137467 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.137957 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.139051 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.139334 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.139411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.142251 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.142258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.142264 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.142302 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.144079 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.144712 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.144715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.144716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.144791 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.145162 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.145166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.145167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.145415 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.145491 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.146182 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.146327 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:47:06.147562 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.134325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34681 (local address 127.0.0.1:57708)
0928 16:47:06.134996 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.135061 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:06.136141 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:06.136563 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:06.136568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:06.137007 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:06.137316 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.137328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.138072 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.138076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.138928 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.138937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.139480 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.139511 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.139704 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.140376 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:06.140392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:06.142004 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:06.144195 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.144205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.144218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.144605 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.144911 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.144916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.144921 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.145084 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.145536 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:06.145543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:06.145886 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 16:47:06.146125 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 16:47:06.146484 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":166,"threads_started":1}
I20260928 16:47:06.147562 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.135303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.30.196.60:41753)
0928 16:47:06.135556 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.135566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.135606 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:47:06.136141 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.136164 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.136398 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.136616 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.137318 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.138357 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.139222 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.139709 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.139807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.142735 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.142743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.142756 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.142782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.144829 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.145409 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.145414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.145415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.145547 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.145833 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.145836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.145917 (+ 81us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.146197 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.146381 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.146896 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.147037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260928 16:47:06.147703 12471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 3
I20260928 16:47:06.149187 12622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:47:06.154065 12546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } attempt: 3
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614026, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:06.160171 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.149381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50152)
0928 16:47:06.149529 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.149547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:06.149629 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:06.150091 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:06.150094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:06.150300 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:06.150423 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.150434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.151549 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.151553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.152335 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.152357 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.152974 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.152977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.153060 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.153588 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:06.153607 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:06.156052 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:06.158513 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.158517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.158519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.158770 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.159074 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.159077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.159079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.159195 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.159634 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:06.159638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:06.159772 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 16:47:06.159877 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:47:06.160005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":8448}
I20260928 16:47:06.160174 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.149898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:33175 (local address 127.0.0.1:45078)
0928 16:47:06.150047 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.150062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:06.150117 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:06.150725 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:06.150728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:06.150866 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:06.150950 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.150954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.151974 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.151977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.152665 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.152674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.153263 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.153267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.153364 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.153755 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:06.153775 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:06.155780 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:06.158367 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.158372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.158377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.158639 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.158937 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.158940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.158941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.159073 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.159570 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:06.159574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:06.159697 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 16:47:06.159858 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 16:47:06.160000 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"spinlock_wait_cycles":9088}
I20260928 16:47:06.160187 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.150035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.30.196.61:33175)
0928 16:47:06.150246 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.150251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.150282 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:47:06.150327 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.150330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.150379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.150459 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.151089 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.151860 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.152788 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.153128 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.153760 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.155921 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.155925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.155928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.155957 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.158234 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.158750 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.158754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.158755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.158855 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.159192 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.159195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.159198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.159455 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.159566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.159912 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.160034 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260928 16:47:06.160737 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.149638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.30.196.60:41753)
0928 16:47:06.149858 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.149862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.149874 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:47:06.149907 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.149912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.149956 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.150534 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.150640 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.151435 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.152474 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.152855 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.152971 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.156202 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.156207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.156212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.156243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.158396 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.158877 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.158880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.158882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.158968 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.159293 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.159296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.159298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.159478 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.159637 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.160466 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.160587 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 16:47:06.161487 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.150332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.30.196.62:34681)
0928 16:47:06.150448 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.150452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.150463 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:47:06.150495 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.150498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.150535 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.150607 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.151348 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.152050 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.153022 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.153304 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.153376 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.156648 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.156652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.156657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.156680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.158487 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.159006 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.159010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.159012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.159103 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.160388 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.160394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.160398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.160607 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.160819 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.161217 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.161329 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:47:06.161592 12622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:47:06.164394 12471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: Bootstrap starting.
I20260928 16:47:06.167404 12471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:06.168458 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.154464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50156)
0928 16:47:06.154717 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.154731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:06.154804 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:06.156579 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:06.156584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:06.156781 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:06.156891 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.156898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.160335 (+ 3437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.160339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.161169 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.161179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.161786 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.161789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.161891 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.162469 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:06.162484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:06.164037 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:06.166759 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.166763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.166765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.167030 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.167337 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.167340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.167342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.167425 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.167901 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:06.167904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:06.167992 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:47:06.168086 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:47:06.168255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 16:47:06.168701 12471 log.cc:824] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:47:06.169468 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.154710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.30.196.60:41753)
0928 16:47:06.156347 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.156358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.156371 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:47:06.156404 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.156409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.156458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.156564 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.157081 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.160233 (+ 3152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.161317 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.161656 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.164374 (+ 2718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.164473 (+ 99us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.164476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.164478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.164494 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.166639 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.167148 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.167151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.167153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.167242 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.167535 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.167538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.167539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.167781 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.169012 (+ 1231us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.169098 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.169242 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":123,"threads_started":1}
Sep 28 16:47:06 dist-test-slave-5rg1 krb5kdc[12390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614026, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:47:06.172812 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: Bootstrap starting.
I20260928 16:47:06.174144 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.163000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.30.196.61:33175)
0928 16:47:06.163107 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.163111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.163318 (+ 207us) server_negotiation.cc:407] Connection header received
0928 16:47:06.163401 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.163406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.163458 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.163550 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.164356 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.165066 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.166350 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.166640 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.166713 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.170506 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.170508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.170510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.170531 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.172171 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.172617 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.172622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.172625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.172713 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.173112 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.173117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.173119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.173343 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.173537 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.173854 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.173997 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:47:06.175724 12546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:06.175997 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.162286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:41753 (local address 127.0.0.1:50158)
0928 16:47:06.164169 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.164180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:06.164286 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:06.165023 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:06.165025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:06.165201 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:06.165290 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.165296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.166327 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.166330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.167138 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:06.167146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.167900 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.167903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:06.167974 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.168364 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:06.168379 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:06.171046 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:06.173446 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.173449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.173451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.173682 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.174829 (+ 1147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:06.174832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:06.174834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:06.174927 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:06.175339 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:06.175342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:06.175438 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 16:47:06.175548 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:47:06.175734 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1823}
I20260928 16:47:06.176488 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.162505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.30.196.60:41753)
0928 16:47:06.163109 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.163121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.164251 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 16:47:06.164776 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.164779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.164820 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.164905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.165410 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.166206 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.167244 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.167700 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.167795 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.171205 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.171209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.171211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.171235 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.173327 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.174328 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.174331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.174333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.174425 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.175034 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.175039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.175040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.175230 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.175332 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.175771 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.175940 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":128,"threads_started":1}
I20260928 16:47:06.176625 12546 log.cc:824] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:06.178563 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: No bootstrap required, opened a new log
I20260928 16:47:06.178574 12471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: No bootstrap required, opened a new log
I20260928 16:47:06.179231 12622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: Bootstrap starting.
I20260928 16:47:06.181385 12622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:06.181989 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:47:06.182689 12471 raft_consensus.cc:348] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.182688 12546 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.182921 12471 raft_consensus.cc:374] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:06.182921 12546 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:06.182981 12546 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2600256598e4cd8a47643921e39bd2c, State: Initialized, Role: FOLLOWER
I20260928 16:47:06.182981 12471 raft_consensus.cc:729] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46c49f4ffda4fc7989cdf1034ec7037, State: Initialized, Role: FOLLOWER
I20260928 16:47:06.183444 12546 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.183444 12471 consensus_queue.cc:260] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.183955 12635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.184181 12635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.184813 12546 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:06.184813 12471 sys_catalog.cc:564] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:06.185791 12622 log.cc:824] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:06.189474 12622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: No bootstrap required, opened a new log
W20260928 16:47:06.193427 12657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:47:06.193512 12657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:47:06.193813 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.193940 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.194734 12622 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.195005 12622 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:06.195060 12622 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1fdcfccd1a4c8283e7627929ac00fa, State: Initialized, Role: FOLLOWER
I20260928 16:47:06.195586 12622 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.197113 12622 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:06.197170 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.185108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.30.196.62:34681)
0928 16:47:06.185245 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.185249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.185262 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:47:06.185304 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.185307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.185350 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.185423 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.186481 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.187296 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.188609 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.188941 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.189031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.190814 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.190818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.190821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.190847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.192529 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.195613 (+ 3084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.195617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.195619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.195739 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.196156 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.196159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.196161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.196398 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.196502 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.196871 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.197021 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 16:47:06.197219 12661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:47:06.197280 12661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:47:06.193917 12403 master_runner.cc:186] Error getting master registration for 127.30.196.60:41753: OK, instance_id { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" instance_seqno: 1790614026122066 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:47:06.202116 12403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:47:06.196709 12478 master_runner.cc:186] Error getting master registration for 127.30.196.60:41753: OK, instance_id { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" instance_seqno: 1790614026122066 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:47:06.203239 12478 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:47:06.204209 12676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:47:06.204293 12676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:47:06.204598 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.204720 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.207898 12553 master_runner.cc:263] No leader master found from master eb1fdcfccd1a4c8283e7627929ac00fa
I20260928 16:47:06.207973 12553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:47:06.209110 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.185253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.30.196.60:41753)
0928 16:47:06.186294 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.186299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.186631 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:47:06.186671 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.186675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.186720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.186788 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.193098 (+ 6310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.194161 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.198324 (+ 4163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.198677 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.198754 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.203973 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.203977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.203980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.204020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.206336 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.206849 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.206853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.206854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.206957 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.207287 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.207291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.207295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.207568 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.207738 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.208717 (+ 979us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.208881 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967}
I20260928 16:47:06.212893 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.195805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.30.196.61:33175)
0928 16:47:06.195928 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:06.195932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:06.197999 (+ 2067us) server_negotiation.cc:407] Connection header received
0928 16:47:06.199546 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:06.199552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:06.199607 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:06.199656 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:06.201825 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.202543 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.204453 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:06.204781 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:06.204867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:06.209539 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:06.209543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:06.209546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:06.209575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:06.211122 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.211538 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.211542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.211544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.211630 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:06.212002 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:06.212006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:06.212007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:06.212198 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:06.212343 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:06.212601 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 16:47:06.212724 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:47:06.358500 12635 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:06.358728 12635 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.359921 12635 leader_election.cc:288] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e46c49f4ffda4fc7989cdf1034ec7037 (127.30.196.62:34681), eb1fdcfccd1a4c8283e7627929ac00fa (127.30.196.60:41753)
I20260928 16:47:06.360857 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2600256598e4cd8a47643921e39bd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" is_pre_election: true
I20260928 16:47:06.361250 12596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2600256598e4cd8a47643921e39bd2c in term 0.
I20260928 16:47:06.361742 12497 leader_election.cc:302] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2600256598e4cd8a47643921e39bd2c, eb1fdcfccd1a4c8283e7627929ac00fa; no voters:
I20260928 16:47:06.362079 12635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:06.362169 12635 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:06.362226 12635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:06.364071 12635 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.364533 12635 leader_election.cc:288] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [CANDIDATE]: Term 1 election: Requested vote from peers e46c49f4ffda4fc7989cdf1034ec7037 (127.30.196.62:34681), eb1fdcfccd1a4c8283e7627929ac00fa (127.30.196.60:41753)
I20260928 16:47:06.365129 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2600256598e4cd8a47643921e39bd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa"
I20260928 16:47:06.365280 12596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:06.366650 12596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2600256598e4cd8a47643921e39bd2c in term 1.
I20260928 16:47:06.367027 12497 leader_election.cc:302] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2600256598e4cd8a47643921e39bd2c, eb1fdcfccd1a4c8283e7627929ac00fa; no voters:
I20260928 16:47:06.367512 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2600256598e4cd8a47643921e39bd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" is_pre_election: true
I20260928 16:47:06.367936 12445 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2600256598e4cd8a47643921e39bd2c in term 0.
W20260928 16:47:06.368335 12575 tablet.cc:2369] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:06.368534 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2600256598e4cd8a47643921e39bd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e46c49f4ffda4fc7989cdf1034ec7037"
I20260928 16:47:06.368656 12445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:06.369784 12635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:06.370064 12635 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [term 1 LEADER]: Becoming Leader. State: Replica: b2600256598e4cd8a47643921e39bd2c, State: Running, Role: LEADER
I20260928 16:47:06.370205 12445 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2600256598e4cd8a47643921e39bd2c in term 1.
I20260928 16:47:06.370571 12635 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } }
I20260928 16:47:06.374606 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2600256598e4cd8a47643921e39bd2c. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.374776 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:06.375262 12680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:06.376528 12680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:06.381609 12596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [term 1 FOLLOWER]: Refusing update from remote peer b2600256598e4cd8a47643921e39bd2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:06.381644 12445 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [term 1 FOLLOWER]: Refusing update from remote peer b2600256598e4cd8a47643921e39bd2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:06.382403 12678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:06.382572 12635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:06.387449 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2600256598e4cd8a47643921e39bd2c. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.387598 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.387568 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2600256598e4cd8a47643921e39bd2c. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.387754 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.392773 12635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.392930 12635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:06.393316 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.393457 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.393464 12681 mvcc.cc:205] Tried to move back new op lower bound from 7334355052051394560 to 7334355052017479680. Current Snapshot: MvccSnapshot[applied={T|T < 7334355052051394560}]
I20260928 16:47:06.393828 12682 mvcc.cc:205] Tried to move back new op lower bound from 7334355052051394560 to 7334355052017479680. Current Snapshot: MvccSnapshot[applied={T|T < 7334355052051394560}]
I20260928 16:47:06.396238 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.396375 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:06.397620 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2600256598e4cd8a47643921e39bd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46c49f4ffda4fc7989cdf1034ec7037" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34681 } } peers { permanent_uuid: "b2600256598e4cd8a47643921e39bd2c" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 33175 } } peers { permanent_uuid: "eb1fdcfccd1a4c8283e7627929ac00fa" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 41753 } } }
I20260928 16:47:06.397763 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:47:06.398540 12680 catalog_manager.cc:1347] Generated new cluster ID: ea086a6bf37d4455869b4e863ed5c26d
I20260928 16:47:06.398625 12680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:06.399822 12683 mvcc.cc:205] Tried to move back new op lower bound from 7334355052051394560 to 7334355052017479680. Current Snapshot: MvccSnapshot[applied={T|T < 7334355052051394560}]
I20260928 16:47:06.419605 12680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:47:06.421028 12680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:06.432946 12680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2600256598e4cd8a47643921e39bd2c: Generated new TSK 0
I20260928 16:47:06.433964 12680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:47:07.194895 12657 catalog_manager.cc:1259] Loaded cluster ID: ea086a6bf37d4455869b4e863ed5c26d
I20260928 16:47:07.194990 12657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: loading cluster ID for follower catalog manager: success
I20260928 16:47:07.196980 12657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: acquiring CA information for follower catalog manager: success
I20260928 16:47:07.198324 12657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e46c49f4ffda4fc7989cdf1034ec7037: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:47:07.205606 12676 catalog_manager.cc:1259] Loaded cluster ID: ea086a6bf37d4455869b4e863ed5c26d
I20260928 16:47:07.205688 12676 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: loading cluster ID for follower catalog manager: success
I20260928 16:47:07.207456 12676 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: acquiring CA information for follower catalog manager: success
I20260928 16:47:07.208711 12676 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb1fdcfccd1a4c8283e7627929ac00fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:47:07.509855 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12403
I20260928 16:47:07.517025 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12478
I20260928 16:47:07.523262 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12553
2026-09-28T16:47:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3222 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:47:07 dist-test-slave-5rg1 krb5kdc[12695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:47:07 dist-test-slave-5rg1 krb5kdc[12695](info): set up 2 sockets
Sep 28 16:47:07 dist-test-slave-5rg1 krb5kdc[12695](info): commencing operation
krb5kdc: starting...
W20260928 16:47:08.580183 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:47:08 dist-test-slave-5rg1 krb5kdc[12695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:47:08.714797 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38469
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:08.829340 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:08.829790 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:08.829885 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:08.834236 12708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:08.834347 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:08.834383 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:08.834426 12708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:08.834458 12708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:08.839968 12708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:08.841090 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:08.842154 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:47:08.851298 12708 server_base.cc:1034] running on GCE node
W20260928 16:47:08.851271 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:08.851526 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:08.851280 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:08.852170 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:08.853227 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:08.854423 12708 hybrid_clock.cc:648] HybridClock initialized: now 1790614028854390 us; error 41 us; skew 500 ppm
Sep 28 16:47:08 dist-test-slave-5rg1 krb5kdc[12695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614028, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:08.858188 12708 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:08.859229 12708 webserver.cc:460] Webserver started at http://127.30.196.62:34965/ using document root <none> and password file <none>
I20260928 16:47:08.859941 12708 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:08.860050 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:08.860323 12708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:08.862582 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "5b5841ee42ca4647a4925515575e62be"
format_stamp: "Formatted at 2026-09-28 16:47:08 on dist-test-slave-5rg1"
server_key: "912a18717a87f4b542d56c9eaa63a9b7"
server_key_iv: "598162550d71854edde420e4d25098fc"
server_key_version: "encryptionkey@0"
I20260928 16:47:08.863180 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "5b5841ee42ca4647a4925515575e62be"
format_stamp: "Formatted at 2026-09-28 16:47:08 on dist-test-slave-5rg1"
server_key: "912a18717a87f4b542d56c9eaa63a9b7"
server_key_iv: "598162550d71854edde420e4d25098fc"
server_key_version: "encryptionkey@0"
I20260928 16:47:08.867043 12708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:47:08.869732 12723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:08.871125 12708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:08.871279 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "5b5841ee42ca4647a4925515575e62be"
format_stamp: "Formatted at 2026-09-28 16:47:08 on dist-test-slave-5rg1"
server_key: "912a18717a87f4b542d56c9eaa63a9b7"
server_key_iv: "598162550d71854edde420e4d25098fc"
server_key_version: "encryptionkey@0"
I20260928 16:47:08.871420 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:08.893996 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:08.897301 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:08.897464 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:08.904902 12708 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38469
I20260928 16:47:08.904989 12775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38469 every 8 connection(s)
I20260928 16:47:08.905928 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:08.908952 12776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:08.912379 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12708
I20260928 16:47:08.912531 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:47:08.912851 31504 external_mini_cluster.cc:1428] Setting key bb00325b50adde9f68ff46b48049839d
I20260928 16:47:08.915345 12776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: Bootstrap starting.
I20260928 16:47:08.917716 12776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:08.918493 12776 log.cc:824] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:08.920784 12776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: No bootstrap required, opened a new log
Sep 28 16:47:08 dist-test-slave-5rg1 krb5kdc[12695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:08.925220 12776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } }
I20260928 16:47:08.925472 12776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:08.925541 12776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5841ee42ca4647a4925515575e62be, State: Initialized, Role: FOLLOWER
I20260928 16:47:08.926074 12776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } }
I20260928 16:47:08.926213 12776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:08.926306 12776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:08.926445 12776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:08.927470 12776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } }
I20260928 16:47:08.927881 12776 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5841ee42ca4647a4925515575e62be; no voters:
I20260928 16:47:08.928275 12776 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:47:08.928365 12781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:08.928582 12781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [term 1 LEADER]: Becoming Leader. State: Replica: 5b5841ee42ca4647a4925515575e62be, State: Running, Role: LEADER
I20260928 16:47:08.929003 12781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } }
I20260928 16:47:08.929174 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:08.914227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.30.196.62:38469)
0928 16:47:08.914715 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:08.914730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:08.914766 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:47:08.915469 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:08.915497 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:08.915921 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:08.916280 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:08.917166 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:08.918273 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:08.919236 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:08.919813 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:08.920009 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:08.922885 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:08.922890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:08.922905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:08.922935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:08.925041 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:08.925662 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:08.925669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:08.925676 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:08.925808 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:08.926374 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:08.926378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:08.926380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:08.926907 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:08.927153 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:08.927481 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 16:47:08.927877 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":155,"threads_started":1}
I20260928 16:47:08.929729 12776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:08.931020 12783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5841ee42ca4647a4925515575e62be. Latest consensus state: current_term: 1 leader_uuid: "5b5841ee42ca4647a4925515575e62be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } } }
I20260928 16:47:08.931161 12783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:08.932377 12782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5841ee42ca4647a4925515575e62be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5841ee42ca4647a4925515575e62be" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38469 } } }
I20260928 16:47:08.932590 12782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:08.932853 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 16:47:08.935084 12796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:47:08.935181 12796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:47:08.935253 12790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:08.936146 12790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:08.941787 12790 catalog_manager.cc:1347] Generated new cluster ID: ced6f1994f3147ed9713a6c3af0576b3
I20260928 16:47:08.941915 12790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:08.952131 12790 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:47:08.953269 12790 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:08.963209 12790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5841ee42ca4647a4925515575e62be: Generated new TSK 0
I20260928 16:47:08.964017 12790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:47:08.989300 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12708
2026-09-28T16:47:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1464 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:47:09 dist-test-slave-5rg1 krb5kdc[12803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:47:09 dist-test-slave-5rg1 krb5kdc[12803](info): set up 2 sockets
Sep 28 16:47:09 dist-test-slave-5rg1 krb5kdc[12803](info): commencing operation
krb5kdc: starting...
W20260928 16:47:10.040560 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:47:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:47:10.176254 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:45397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:10.292819 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:10.293164 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:10.293244 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:10.297509 12816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:10.297623 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:10.297662 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:10.297698 12816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:10.297729 12816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:10.303455 12816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:45397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:10.304636 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:10.305676 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:10.315408 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:10.315472 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:10.315553 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:10.315651 12816 server_base.cc:1034] running on GCE node
I20260928 16:47:10.316315 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:10.317585 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:10.318774 12816 hybrid_clock.cc:648] HybridClock initialized: now 1790614030318756 us; error 35 us; skew 500 ppm
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:10.329180 12816 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:10.330176 12816 webserver.cc:460] Webserver started at http://127.30.196.62:41447/ using document root <none> and password file <none>
I20260928 16:47:10.330787 12816 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:10.330892 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:10.331147 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:10.333443 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "65c54f2c027b439c9c0702d3d71c6023"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "7e92cdcc23d433c69f1af465bbadb9be"
server_key_iv: "c3eac68d86b59661eb7bb39d1ef29d2e"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.334229 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "65c54f2c027b439c9c0702d3d71c6023"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "7e92cdcc23d433c69f1af465bbadb9be"
server_key_iv: "c3eac68d86b59661eb7bb39d1ef29d2e"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.338480 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:47:10.341446 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.343051 12816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:47:10.343238 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "65c54f2c027b439c9c0702d3d71c6023"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "7e92cdcc23d433c69f1af465bbadb9be"
server_key_iv: "c3eac68d86b59661eb7bb39d1ef29d2e"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.343415 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:10.359017 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:10.362437 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:10.362685 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:10.370293 12816 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:45397
I20260928 16:47:10.370611 12883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:45397 every 8 connection(s)
I20260928 16:47:10.371363 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:10.374029 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12816
I20260928 16:47:10.374195 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:47:10.374383 12884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:10.374531 31504 external_mini_cluster.cc:1428] Setting key 54b8e7e609fe19ecb530de4f91879394
I20260928 16:47:10.382478 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap starting.
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:10.385402 12884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:10.386196 12884 log.cc:824] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:10.388376 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: No bootstrap required, opened a new log
I20260928 16:47:10.390357 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.376065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.30.196.62:45397)
0928 16:47:10.376553 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.376568 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.376604 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:47:10.377596 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.377616 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.377945 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.378371 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:10.379333 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.380466 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.381691 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.382166 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.382287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.385193 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:10.385198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:10.385212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:10.385241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:10.387294 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.387808 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.387818 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.387823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.387935 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.388257 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.388263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.388264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.388731 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:10.388968 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:10.389294 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 16:47:10.389616 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":161,"threads_started":1}
I20260928 16:47:10.392746 12884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:10.393069 12884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:10.393172 12884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Initialized, Role: FOLLOWER
I20260928 16:47:10.393615 12884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:10.393774 12884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:10.393875 12884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:10.393983 12884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:10.394984 12884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:10.395356 12884 leader_election.cc:302] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c54f2c027b439c9c0702d3d71c6023; no voters:
I20260928 16:47:10.395753 12884 leader_election.cc:288] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:47:10.395911 12889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:10.396207 12889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 LEADER]: Becoming Leader. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Running, Role: LEADER
I20260928 16:47:10.396543 12889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:10.397187 12884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:10.398473 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c54f2c027b439c9c0702d3d71c6023. Latest consensus state: current_term: 1 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:10.398624 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:10.399001 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:10.399106 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:10.399061 12898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:10.402139 12898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:10.407809 12898 catalog_manager.cc:1347] Generated new cluster ID: 3af27c6dad6b43fc9ba11349ccb66924
I20260928 16:47:10.407896 12898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:10.416109 12898 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:47:10.417147 12898 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:10.421413 12898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Generated new TSK 0
I20260928 16:47:10.422155 12898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:47:10.456934 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:10.577946 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:10.578223 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:10.578269 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:10.582525 12910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:10.582597 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:10.582659 12910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:10.588124 12910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:10.589200 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:10.590173 12910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:10.600713 12918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:10.600776 12915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:10.600890 12910 server_base.cc:1034] running on GCE node
W20260928 16:47:10.600713 12916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:10.601519 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:10.602273 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:10.603493 12910 hybrid_clock.cc:648] HybridClock initialized: now 1790614030603476 us; error 36 us; skew 500 ppm
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:10.606346 12910 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:10.607307 12910 webserver.cc:460] Webserver started at http://127.30.196.1:39231/ using document root <none> and password file <none>
I20260928 16:47:10.608052 12910 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:10.608153 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:10.608402 12910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:10.610634 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "0e6495fa75eb4137a8b328c4744b17ee"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "af3d8076be624a1a4c924b118f2e8327"
server_key_iv: "d4dbf95808421d2bed73778dab29f0d9"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.611361 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "0e6495fa75eb4137a8b328c4744b17ee"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "af3d8076be624a1a4c924b118f2e8327"
server_key_iv: "d4dbf95808421d2bed73778dab29f0d9"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.615511 12910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:47:10.618261 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.619536 12910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:10.619722 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "0e6495fa75eb4137a8b328c4744b17ee"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "af3d8076be624a1a4c924b118f2e8327"
server_key_iv: "d4dbf95808421d2bed73778dab29f0d9"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.619865 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:10.634644 12910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:10.638268 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:10.638545 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:10.639223 12910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:10.640232 12910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:10.640318 12910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.640389 12910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:10.640424 12910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.650184 12910 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:33377
I20260928 16:47:10.650204 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:33377 every 8 connection(s)
I20260928 16:47:10.651160 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:10.654345 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 12910
I20260928 16:47:10.654500 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:47:10.654838 31504 external_mini_cluster.cc:1428] Setting key 8517aa5c9448603066b8613ba504a90d
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:47:10.667594 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.653295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:60011 (local address 127.30.196.62:45397)
0928 16:47:10.653504 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.653509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.654716 (+ 1207us) server_negotiation.cc:407] Connection header received
0928 16:47:10.656336 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.656339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.656393 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.656520 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:10.657831 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.658377 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.659641 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.659950 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.660081 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.663151 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:10.663155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:10.663157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:10.663184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:10.664958 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.665588 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.665592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.665594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.665685 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.666286 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.666289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.666291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.666540 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:10.666669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:10.667242 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 16:47:10.667401 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:47:10.668574 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.653697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:60011)
0928 16:47:10.654507 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:10.654590 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:10.656104 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:10.656706 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:10.656713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:10.657143 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:10.657660 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:10.657675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.658552 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.658555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:10.659464 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:10.659474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.660134 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.660138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:10.660322 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.660999 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:10.661018 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:10.662925 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:10.665107 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:10.665114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:10.665133 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:10.665458 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:10.665795 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:10.665799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:10.665801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:10.666127 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:10.666669 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:10.666678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:10.666979 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 16:47:10.667199 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 16:47:10.667570 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:47:10.670030 13041 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:10.670313 13041 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:10.670954 13041 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:10.672868 12848 ts_manager.cc:194] Registered new tserver with Master: 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377)
I20260928 16:47:10.674079 12848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:60011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:47:10.688436 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:10.805794 13046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:10.806079 13046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:10.806126 13046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:10.810513 13046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:10.810588 13046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:10.810649 13046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:10.816358 13046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:10.817516 13046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:10.818624 13046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:10.828711 13051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:10.828799 13052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:10.828707 13054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:10.828989 13046 server_base.cc:1034] running on GCE node
I20260928 16:47:10.829557 13046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:10.830255 13046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:10.831482 13046 hybrid_clock.cc:648] HybridClock initialized: now 1790614030831466 us; error 25 us; skew 500 ppm
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:10.834928 13046 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:10.836107 13046 webserver.cc:460] Webserver started at http://127.30.196.2:46213/ using document root <none> and password file <none>
I20260928 16:47:10.836727 13046 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:10.836833 13046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:10.837057 13046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:10.839274 13046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "4c74cd53a4414018912d79484cc08ac0"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "0c191fabb962b5f9975eb85c1da8ef8a"
server_key_iv: "fa321aa064325b88836b3e2efdd50fc1"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.839958 13046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "4c74cd53a4414018912d79484cc08ac0"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "0c191fabb962b5f9975eb85c1da8ef8a"
server_key_iv: "fa321aa064325b88836b3e2efdd50fc1"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.844027 13046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:10.846623 13061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.848037 13046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:10.848186 13046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "4c74cd53a4414018912d79484cc08ac0"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "0c191fabb962b5f9975eb85c1da8ef8a"
server_key_iv: "fa321aa064325b88836b3e2efdd50fc1"
server_key_version: "encryptionkey@0"
I20260928 16:47:10.848313 13046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:10.879453 13046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:10.883126 13046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:10.883396 13046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:10.884191 13046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:10.885130 13046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:10.885216 13046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.885289 13046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:10.885341 13046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:10.894975 13046 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38559
I20260928 16:47:10.894996 13175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38559 every 8 connection(s)
I20260928 16:47:10.896037 13046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:10.896633 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13046
I20260928 16:47:10.896764 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:47:10.897126 31504 external_mini_cluster.cc:1428] Setting key 2633358193489fd3bd7492763782c5a0
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Sep 28 16:47:10 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614030, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:47:10.911804 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.898338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:47963 (local address 127.30.196.62:45397)
0928 16:47:10.898483 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.898487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.899614 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 16:47:10.901106 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.901109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.901171 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.901254 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:10.902601 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.903149 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.904410 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.904740 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.904872 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.907457 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:10.907461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:10.907464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:10.907494 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:10.909112 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.909682 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.909686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.909688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.909774 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.910255 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.910262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.910267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.910554 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:10.910649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:10.911457 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 16:47:10.911602 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:47:10.912781 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.898877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:47963)
0928 16:47:10.899417 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:10.899506 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:10.900872 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:10.901428 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:10.901435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:10.901831 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:10.902442 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:10.902458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.903292 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.903295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:10.904273 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:10.904283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.904917 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.904921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:10.905097 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.905756 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:10.905773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:10.907270 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:10.909248 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:10.909256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:10.909273 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:10.909561 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:10.909883 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:10.909886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:10.909888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:10.910042 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:10.910684 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:10.910690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:10.911108 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 16:47:10.911349 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 16:47:10.911799 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
I20260928 16:47:10.914150 13176 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:10.914424 13176 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:10.915053 13176 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:10.916296 12848 ts_manager.cc:194] Registered new tserver with Master: 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
I20260928 16:47:10.916904 12848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:47963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:47:10.928292 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:11.044267 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:11.044543 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:11.044586 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:11.048796 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:11.048883 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:11.048990 13181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:11.054536 13181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:11.055753 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:11.056787 13181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:11.066974 13189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.067092 13181 server_base.cc:1034] running on GCE node
W20260928 16:47:11.067129 13186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.066968 13187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.067749 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:11.068421 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:11.069630 13181 hybrid_clock.cc:648] HybridClock initialized: now 1790614031069598 us; error 44 us; skew 500 ppm
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:11.072553 13181 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:11.073486 13181 webserver.cc:460] Webserver started at http://127.30.196.3:43899/ using document root <none> and password file <none>
I20260928 16:47:11.074118 13181 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:11.074222 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:11.074456 13181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:11.076659 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "1bbd392678684d85ba071f9685e05568"
format_stamp: "Formatted at 2026-09-28 16:47:11 on dist-test-slave-5rg1"
server_key: "a19daaeb43cc2e686a20ddb6b567a0ba"
server_key_iv: "c781def01ab9675a26537735770b72f0"
server_key_version: "encryptionkey@0"
I20260928 16:47:11.077227 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "1bbd392678684d85ba071f9685e05568"
format_stamp: "Formatted at 2026-09-28 16:47:11 on dist-test-slave-5rg1"
server_key: "a19daaeb43cc2e686a20ddb6b567a0ba"
server_key_iv: "c781def01ab9675a26537735770b72f0"
server_key_version: "encryptionkey@0"
I20260928 16:47:11.081082 13181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:47:11.083758 13196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:11.085096 13181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:11.085232 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "1bbd392678684d85ba071f9685e05568"
format_stamp: "Formatted at 2026-09-28 16:47:11 on dist-test-slave-5rg1"
server_key: "a19daaeb43cc2e686a20ddb6b567a0ba"
server_key_iv: "c781def01ab9675a26537735770b72f0"
server_key_version: "encryptionkey@0"
I20260928 16:47:11.085358 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:11.097018 13181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:11.100668 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:11.100890 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:11.101509 13181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:11.102372 13181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:11.102449 13181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:11.102525 13181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:11.102558 13181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:11.112536 13181 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:38673
I20260928 16:47:11.112555 13310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:38673 every 8 connection(s)
I20260928 16:47:11.113503 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:11.115538 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13181
I20260928 16:47:11.115648 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:47:11.115975 31504 external_mini_cluster.cc:1428] Setting key 8bb780c169e60442400af79c9f4d8a90
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:11.129391 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.115436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:47501 (local address 127.30.196.62:45397)
0928 16:47:11.115611 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.115616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.116855 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 16:47:11.118893 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.118897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.118947 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.119023 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:11.120409 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.120936 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.122271 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.122502 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.122585 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.125646 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:11.125649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:11.125651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:11.125677 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:11.127092 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.127624 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.127627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.127628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.127786 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.128173 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.128176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.128177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.128367 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:11.128442 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.129064 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.129227 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:47:11.130303 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.116029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:47501)
0928 16:47:11.116661 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.116711 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.118648 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.119216 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.119224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.119701 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:11.120261 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.120276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.121059 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.121062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.122141 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.122153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.122682 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.122686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.122858 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.123747 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:11.123768 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:11.125473 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:11.127215 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:11.127220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:11.127234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:11.127513 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:11.127872 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:11.127874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:11.127875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:11.128075 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:11.128465 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:11.128471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:11.128736 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.128917 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.129312 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":138,"threads_started":1}
I20260928 16:47:11.131534 13311 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:11.131829 13311 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:11.132362 13311 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:11.133625 12848 ts_manager.cc:194] Registered new tserver with Master: 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673)
I20260928 16:47:11.134234 12848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:47501
I20260928 16:47:11.142755 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:11.153683 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.145101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.30.196.62:45397)
0928 16:47:11.145285 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.145290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.145304 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:47:11.145405 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.145408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.145457 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.145560 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:11.146302 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.146839 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.147841 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.148074 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.148207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.150057 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:11.150060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:11.150062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:11.150090 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:11.151760 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.152265 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.152267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.152269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.152345 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.152656 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.152660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.152661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.152966 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:11.153128 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.153369 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.153506 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:47:11.157166 12848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:11.160404 12848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:11.176075 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.169868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.30.196.2:38559)
0928 16:47:11.170179 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.170191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.170313 (+ 122us) server_negotiation.cc:407] Connection header received
0928 16:47:11.170875 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.170880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.171040 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.171181 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.171482 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.172780 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.174720 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175332 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175474 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175624 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.175826 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.175932 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":130,"threads_started":1}
I20260928 16:47:11.176067 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.169275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.0.0.1:42298)
0928 16:47:11.170154 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.170209 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.170352 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.171215 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.171220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.171240 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.171349 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.171354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.172956 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.172959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.174564 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.174576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175474 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.175589 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175627 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.175767 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.175907 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":160,"thread_start_us":107,"threads_started":1}
I20260928 16:47:11.176321 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.170084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:38673 (local address 127.0.0.1:44946)
0928 16:47:11.170597 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.170611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.170687 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.171957 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.171960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.171978 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.172065 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.172070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.173106 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.173109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.174333 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.174342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175727 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.175824 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175909 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.175985 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.176062 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":236,"thread_start_us":89,"threads_started":1}
I20260928 16:47:11.176401 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.170544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.30.196.3:38673)
0928 16:47:11.171489 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.171500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.171523 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:47:11.171599 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.171604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.171826 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.171978 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.172157 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.172967 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.174472 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175338 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175455 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175623 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.176136 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.176243 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":90,"threads_started":1}
I20260928 16:47:11.176868 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.169457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.30.196.1:33377)
0928 16:47:11.169728 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.169736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.170558 (+ 822us) server_negotiation.cc:407] Connection header received
0928 16:47:11.170809 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.170817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.171041 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.171166 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.171619 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.172780 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.174702 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175485 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175580 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175725 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.176048 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.176716 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20260928 16:47:11.177318 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.169275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.0.0.1:50190)
0928 16:47:11.170443 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.170458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.170641 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.171367 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.171370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.171388 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.171486 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.171493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.172921 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.172925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.174564 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.174576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.175774 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.175777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.175879 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.175920 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.177096 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.177183 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":266,"thread_start_us":100,"threads_started":1}
I20260928 16:47:11.179363 13244 tablet_service.cc:1467] Processing CreateTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 (DEFAULT_TABLE table=test-table [id=1a3b24a7b59e438ab99278df04a2b844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:11.179570 12974 tablet_service.cc:1467] Processing CreateTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 (DEFAULT_TABLE table=test-table [id=1a3b24a7b59e438ab99278df04a2b844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:11.180567 13244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01f6c289d5f485c8631865c1fa7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:11.180640 12974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01f6c289d5f485c8631865c1fa7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:11.181978 13110 tablet_service.cc:1467] Processing CreateTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 (DEFAULT_TABLE table=test-table [id=1a3b24a7b59e438ab99278df04a2b844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:11.183101 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01f6c289d5f485c8631865c1fa7ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:11.186981 13328 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Bootstrap starting.
I20260928 16:47:11.188565 13329 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap starting.
I20260928 16:47:11.189510 13328 tablet_bootstrap.cc:654] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:11.190835 13328 log.cc:824] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:11.190867 13329 tablet_bootstrap.cc:654] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:11.191464 13330 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Bootstrap starting.
I20260928 16:47:11.191731 13329 log.cc:824] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:11.193226 13328 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: No bootstrap required, opened a new log
I20260928 16:47:11.193444 13328 ts_tablet_manager.cc:1397] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:47:11.193809 13329 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: No bootstrap required, opened a new log
I20260928 16:47:11.194018 13329 ts_tablet_manager.cc:1397] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:47:11.194741 13330 tablet_bootstrap.cc:654] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:11.195588 13330 log.cc:824] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:11.197772 13330 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: No bootstrap required, opened a new log
I20260928 16:47:11.197664 13328 raft_consensus.cc:348] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.197953 13328 raft_consensus.cc:374] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:11.197971 13330 ts_tablet_manager.cc:1397] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 16:47:11.198019 13328 raft_consensus.cc:729] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbd392678684d85ba071f9685e05568, State: Initialized, Role: FOLLOWER
I20260928 16:47:11.197837 13329 raft_consensus.cc:348] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.198091 13329 raft_consensus.cc:374] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:11.198153 13329 raft_consensus.cc:729] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6495fa75eb4137a8b328c4744b17ee, State: Initialized, Role: FOLLOWER
I20260928 16:47:11.198608 13328 consensus_queue.cc:260] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.198602 13329 consensus_queue.cc:260] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.199656 13328 ts_tablet_manager.cc:1428] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:47:11.199635 13329 ts_tablet_manager.cc:1428] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:47:11.200037 13311 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:11.200110 13041 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:11.202059 13330 raft_consensus.cc:348] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.202339 13330 raft_consensus.cc:374] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:11.202380 13330 raft_consensus.cc:729] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cd53a4414018912d79484cc08ac0, State: Initialized, Role: FOLLOWER
I20260928 16:47:11.202862 13330 consensus_queue.cc:260] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.203580 13176 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:11.203966 13330 ts_tablet_manager.cc:1428] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260928 16:47:11.294090 13335 raft_consensus.cc:482] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:11.294303 13335 raft_consensus.cc:504] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.295496 13335 leader_election.cc:288] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
I20260928 16:47:11.299530 13336 raft_consensus.cc:482] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:11.299930 13336 raft_consensus.cc:504] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.300475 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.295974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:41801 (local address 127.30.196.1:33377)
0928 16:47:11.296091 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.296096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.296115 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:47:11.296304 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.296306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.296352 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.296458 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.296824 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.297528 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.299086 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.299772 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.299890 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.299943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.300260 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.300349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:47:11.300480 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.295866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.30.196.3:41801)
0928 16:47:11.295990 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.296009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.296124 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.296493 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.296499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.296527 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.296693 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.296701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.297657 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.297660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.298970 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.298983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.300007 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.300011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.300128 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.300151 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.300235 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.300311 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":4,"spinlock_wait_cycles":2944}
I20260928 16:47:11.300786 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.295942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55387 (local address 127.30.196.2:38559)
0928 16:47:11.296091 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.296096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.296600 (+ 504us) server_negotiation.cc:407] Connection header received
0928 16:47:11.296720 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.296723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.296808 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.296896 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.297267 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.298047 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.299533 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.300163 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.300273 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.300316 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.300618 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.300703 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:47:11.301129 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.295866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.30.196.3:55387)
0928 16:47:11.296453 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.296470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.296579 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.296983 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.296985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.297003 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.297107 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.297121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.298190 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.298194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.299375 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.299405 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.300373 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.300377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.300467 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.300490 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.300951 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.301026 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":123,"thread_start_us":148,"threads_started":1}
I20260928 16:47:11.301252 12994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6495fa75eb4137a8b328c4744b17ee" is_pre_election: true
I20260928 16:47:11.301388 13336 leader_election.cc:288] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377), 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673)
I20260928 16:47:11.301548 12994 raft_consensus.cc:2454] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 0.
I20260928 16:47:11.301692 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cd53a4414018912d79484cc08ac0" is_pre_election: true
I20260928 16:47:11.301887 13130 raft_consensus.cc:2454] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 0.
I20260928 16:47:11.302407 13198 leader_election.cc:302] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee, 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0; no voters:
I20260928 16:47:11.302712 13335 raft_consensus.cc:2781] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:11.302847 13335 raft_consensus.cc:482] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:11.302915 13335 raft_consensus.cc:3037] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:11.309783 13335 raft_consensus.cc:504] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.310468 13335 leader_election.cc:288] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 election: Requested vote from peers 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
I20260928 16:47:11.310973 12994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6495fa75eb4137a8b328c4744b17ee"
I20260928 16:47:11.311161 12994 raft_consensus.cc:3037] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:11.312542 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cd53a4414018912d79484cc08ac0"
I20260928 16:47:11.312739 13130 raft_consensus.cc:3037] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:11.313344 12994 raft_consensus.cc:2454] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 1.
I20260928 16:47:11.313876 13199 leader_election.cc:302] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee, 1bbd392678684d85ba071f9685e05568; no voters:
I20260928 16:47:11.314165 13335 raft_consensus.cc:2781] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:11.314461 13335 raft_consensus.cc:686] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbd392678684d85ba071f9685e05568, State: Running, Role: LEADER
I20260928 16:47:11.315040 13335 consensus_queue.cc:237] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.315488 13130 raft_consensus.cc:2454] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 1.
I20260928 16:47:11.317183 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.302103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:38673 (local address 127.30.196.2:60607)
0928 16:47:11.308157 (+ 6054us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.308177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.308301 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.311485 (+ 3184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.311492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.311521 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.311784 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.311802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.313612 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.313616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.315652 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.315706 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.316707 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.316710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.316821 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.316862 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.316938 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.317015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5948,"mutex_wait_us":97,"thread_start_us":123,"threads_started":1}
I20260928 16:47:11.317224 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.302567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:60607 (local address 127.30.196.3:38673)
0928 16:47:11.302687 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.302691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.311180 (+ 8489us) server_negotiation.cc:407] Connection header received
0928 16:47:11.311253 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.311256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.311324 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.311414 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.312190 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.313314 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.315833 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.316546 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.316695 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.316754 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.317010 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.317095 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:47:11.318218 13265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "4c74cd53a4414018912d79484cc08ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbd392678684d85ba071f9685e05568" is_pre_election: true
I20260928 16:47:11.319880 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.302103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.30.196.2:36409)
0928 16:47:11.302255 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.302271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.302375 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.311366 (+ 8991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.311372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.311401 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.311560 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.311570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.316297 (+ 4727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.316302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.317573 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.317586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.319340 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.319346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.319479 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.319502 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.319618 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.319734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 16:47:11.319913 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.302142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:36409 (local address 127.30.196.1:33377)
0928 16:47:11.307372 (+ 5230us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.307379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.307404 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:47:11.307470 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.307475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.307548 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.307685 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.311730 (+ 4045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.312651 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.317719 (+ 5068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.318501 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.318646 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.318702 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.319642 (+ 940us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.319770 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5093}
I20260928 16:47:11.320106 12848 catalog_manager.cc:5495] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bbd392678684d85ba071f9685e05568 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "1bbd392678684d85ba071f9685e05568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } health_report { overall_health: HEALTHY } } }
I20260928 16:47:11.321687 12994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01f6c289d5f485c8631865c1fa7ef76" candidate_uuid: "4c74cd53a4414018912d79484cc08ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6495fa75eb4137a8b328c4744b17ee" is_pre_election: true
I20260928 16:47:11.321949 12994 raft_consensus.cc:2379] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c74cd53a4414018912d79484cc08ac0 in current term 1: Already voted for candidate 1bbd392678684d85ba071f9685e05568 in this term.
I20260928 16:47:11.322578 13064 leader_election.cc:302] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c74cd53a4414018912d79484cc08ac0; no voters: 0e6495fa75eb4137a8b328c4744b17ee, 1bbd392678684d85ba071f9685e05568
I20260928 16:47:11.322928 13336 raft_consensus.cc:2726] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:47:11.337738 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.333803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.30.196.3:38673)
0928 16:47:11.333961 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.333966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.333981 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:47:11.334092 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.334094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.334149 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.334274 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:11.334540 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.335053 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.336031 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.336398 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.336523 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.336863 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:11.337315 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:11.337474 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.337536 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.337594 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:47:11.344144 12994 raft_consensus.cc:1261] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Refusing update from remote peer 1bbd392678684d85ba071f9685e05568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:11.344167 13130 raft_consensus.cc:1261] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Refusing update from remote peer 1bbd392678684d85ba071f9685e05568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:11.345331 13339 consensus_queue.cc:1035] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:11.345656 13335 consensus_queue.cc:1035] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:11.356746 13342 mvcc.cc:205] Tried to move back new op lower bound from 7334355072377303040 to 7334355072271790080. Current Snapshot: MvccSnapshot[applied={T|T < 7334355072377303040}]
I20260928 16:47:11.357820 13343 mvcc.cc:205] Tried to move back new op lower bound from 7334355072377303040 to 7334355072271790080. Current Snapshot: MvccSnapshot[applied={T|T < 7334355072377303040}]
I20260928 16:47:11.361522 13344 mvcc.cc:205] Tried to move back new op lower bound from 7334355072377303040 to 7334355072271790080. Current Snapshot: MvccSnapshot[applied={T|T < 7334355072377303040}]
I20260928 16:47:11.368191 12848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:11.368429 12848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:11.376032 12848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Sending DeleteTablet for 3 replicas of tablet c01f6c289d5f485c8631865c1fa7ef76
I20260928 16:47:11.377332 12974 tablet_service.cc:1514] Processing DeleteTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:11 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:50190
I20260928 16:47:11.377446 13110 tablet_service.cc:1514] Processing DeleteTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:11 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:42298
I20260928 16:47:11.377928 13244 tablet_service.cc:1514] Processing DeleteTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:11 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260928 16:47:11.378239 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12910
I20260928 16:47:11.379122 13353 tablet_replica.cc:330] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: stopping tablet replica
I20260928 16:47:11.379520 13353 raft_consensus.cc:2229] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:11.380111 13353 raft_consensus.cc:2258] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:11.388927 13352 tablet_replica.cc:330] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: stopping tablet replica
I20260928 16:47:11.389403 13352 raft_consensus.cc:2229] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:11.389912 13352 raft_consensus.cc:2258] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:11.391566 13353 ts_tablet_manager.cc:1905] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:11.393610 13352 ts_tablet_manager.cc:1905] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:11.394454 13353 ts_tablet_manager.cc:1918] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:11.394572 13353 log.cc:1196] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/c01f6c289d5f485c8631865c1fa7ef76
I20260928 16:47:11.394927 13353 ts_tablet_manager.cc:1939] T c01f6c289d5f485c8631865c1fa7ef76 P 1bbd392678684d85ba071f9685e05568: Deleting consensus metadata
W20260928 16:47:11.395969 12834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:33377 (error 108)
I20260928 16:47:11.396783 12833 catalog_manager.cc:4841] TS 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): tablet c01f6c289d5f485c8631865c1fa7ef76 (table test-table [id=1a3b24a7b59e438ab99278df04a2b844]) successfully deleted
I20260928 16:47:11.396956 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13046
I20260928 16:47:11.404677 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.404304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.0.0.1:50194)
0928 16:47:11.404504 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.404580 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 16:47:11.404883 12834 catalog_manager.cc:4584] TS 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c01f6c289d5f485c8631865c1fa7ef76: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33377: connect: Connection refused (error 111)
I20260928 16:47:11.406963 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13181
I20260928 16:47:11.408614 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.406996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.0.0.1:42300)
0928 16:47:11.408443 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.408512 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1351}
I20260928 16:47:11.413511 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 12816
I20260928 16:47:11.420135 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--webserver_interface=127.30.196.62
--webserver_port=41447
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:45397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:11.550871 13355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:11.551221 13355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:11.551266 13355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:11.555495 13355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:11.555570 13355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:11.555590 13355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:11.555608 13355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:11.555624 13355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:11.561254 13355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:45397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:11.562466 13355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:11.563584 13355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:11.577414 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.577414 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.577414 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.577888 13355 server_base.cc:1034] running on GCE node
I20260928 16:47:11.578583 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:11.579895 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:11.581089 13355 hybrid_clock.cc:648] HybridClock initialized: now 1790614031581069 us; error 31 us; skew 500 ppm
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:11.584152 13355 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:11.585170 13355 webserver.cc:460] Webserver started at http://127.30.196.62:41447/ using document root <none> and password file <none>
I20260928 16:47:11.585784 13355 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:11.585866 13355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:11.590029 13355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:11.592391 13371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:11.593722 13355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:11.593883 13355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "65c54f2c027b439c9c0702d3d71c6023"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "7e92cdcc23d433c69f1af465bbadb9be"
server_key_iv: "c3eac68d86b59661eb7bb39d1ef29d2e"
server_key_version: "encryptionkey@0"
I20260928 16:47:11.594512 13355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:11.606813 13355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:11.610664 13355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:11.610936 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:11.618716 13355 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:45397
I20260928 16:47:11.619045 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:45397 every 8 connection(s)
I20260928 16:47:11.620018 13355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:11.624601 13425 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:11.626375 13425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap starting.
I20260928 16:47:11.627110 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13355
I20260928 16:47:11.627578 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:33377
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=39231
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:11.644394 13425 log.cc:824] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:11.654114 13425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:11.654618 13425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap complete.
I20260928 16:47:11.658999 13425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:11.659446 13425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Initialized, Role: FOLLOWER
I20260928 16:47:11.660046 13425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:11.660180 13425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:11.660290 13425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:11.660387 13425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:47:11.662555 13425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:11.662966 13425 leader_election.cc:302] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c54f2c027b439c9c0702d3d71c6023; no voters:
I20260928 16:47:11.663375 13425 leader_election.cc:288] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:47:11.663452 13429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:47:11.663704 13429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 LEADER]: Becoming Leader. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Running, Role: LEADER
I20260928 16:47:11.664023 13429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:11.664777 13425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:11.665081 13431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:11.665203 13431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:11.665427 13430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c54f2c027b439c9c0702d3d71c6023. Latest consensus state: current_term: 2 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:11.665529 13430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:11.665599 13433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:11.670733 13433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:11.671528 13433 catalog_manager.cc:1259] Loaded cluster ID: 3af27c6dad6b43fc9ba11349ccb66924
I20260928 16:47:11.671597 13433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:11.673059 13433 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 16:47:11.678015 13433 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 16:47:11.684057 13433 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:11.684931 13433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Loaded TSK: 0
I20260928 16:47:11.686277 13433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:47:11.751066 13427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:11.751410 13427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:11.751497 13427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:11.755983 13427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:11.756088 13427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:11.756183 13427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:11.761806 13427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:33377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=39231
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:11.762928 13427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:11.764132 13427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:11.774935 13452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.774935 13453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.775148 13427 server_base.cc:1034] running on GCE node
W20260928 16:47:11.775352 13455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.775868 13427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:11.776615 13427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:11.777827 13427 hybrid_clock.cc:648] HybridClock initialized: now 1790614031777811 us; error 35 us; skew 500 ppm
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:11.781059 13427 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:11.782200 13427 webserver.cc:460] Webserver started at http://127.30.196.1:39231/ using document root <none> and password file <none>
I20260928 16:47:11.782828 13427 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:11.782910 13427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:11.787425 13427 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:11.789896 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:11.791324 13427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:11.791443 13427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "0e6495fa75eb4137a8b328c4744b17ee"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "af3d8076be624a1a4c924b118f2e8327"
server_key_iv: "d4dbf95808421d2bed73778dab29f0d9"
server_key_version: "encryptionkey@0"
I20260928 16:47:11.792145 13427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:11.807363 13427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:11.810812 13427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:11.811048 13427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:11.811774 13427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:11.812950 13469 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:47:11.817318 13427 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:47:11.817387 13427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 16:47:11.817447 13427 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:47:11.819206 13427 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:47:11.819268 13427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:47:11.819526 13469 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap starting.
I20260928 16:47:11.830956 13427 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:33377
I20260928 16:47:11.831035 13577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:33377 every 8 connection(s)
I20260928 16:47:11.832312 13427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:11.835415 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13427
I20260928 16:47:11.836018 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:38559
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=46213
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:11.842015 13469 log.cc:824] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:11.850747 13469 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:11.851167 13469 tablet_bootstrap.cc:492] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap complete.
I20260928 16:47:11.852607 13469 ts_tablet_manager.cc:1397] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.015s
I20260928 16:47:11.855581 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.834961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:40403)
0928 16:47:11.836138 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.836226 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.837904 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.839857 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.839864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.840278 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:11.840793 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.840810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.842760 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.842767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.844582 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.844600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.845662 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.845670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.845935 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.846726 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:11.846754 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:11.848943 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:11.851897 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:11.851912 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:11.851928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:11.852355 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:11.852823 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:11.852826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:11.852828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:11.853004 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:11.853755 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:11.853761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:11.854048 (+ 287us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.854237 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.854589 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":184,"threads_started":1}
I20260928 16:47:11.856381 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.834917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:40403 (local address 127.30.196.62:45397)
0928 16:47:11.835752 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.835779 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.836372 (+ 593us) server_negotiation.cc:407] Connection header received
0928 16:47:11.838546 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.838572 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.838981 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.839265 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:11.841020 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.842548 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.844759 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.845468 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.845654 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.849240 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:11.849251 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:11.849271 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:11.849312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:11.851762 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.852504 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.852510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.852517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.852705 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.853140 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.853144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.853146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.853622 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:11.853763 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.854590 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.854980 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":183,"threads_started":1}
I20260928 16:47:11.856125 13469 raft_consensus.cc:348] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.856739 13469 raft_consensus.cc:729] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6495fa75eb4137a8b328c4744b17ee, State: Initialized, Role: FOLLOWER
I20260928 16:47:11.857317 13469 consensus_queue.cc:260] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } }
I20260928 16:47:11.858343 13469 ts_tablet_manager.cc:1428] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 16:47:11.859184 13578 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:11.859516 13578 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:11.860181 13578 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:11.863160 13389 ts_manager.cc:194] Registered new tserver with Master: 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377)
I20260928 16:47:11.867643 13389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:40403
I20260928 16:47:11.870525 13578 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:11.873953 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.868797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.30.196.1:33377)
0928 16:47:11.869188 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:11.869201 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:11.869511 (+ 310us) server_negotiation.cc:407] Connection header received
0928 16:47:11.869749 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:11.869755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:11.869933 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:11.870109 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:11.870408 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.871378 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.872707 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.873254 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.873401 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.873543 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.873693 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.873788 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":154,"threads_started":1}
I20260928 16:47:11.873953 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:11.866911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.0.0.1:50208)
0928 16:47:11.869349 (+ 2438us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:11.869403 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:11.869581 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:11.870116 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:11.870121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:11.870144 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:11.870274 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.870282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.871503 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.871506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.872582 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:11.872591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.873405 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.873408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:11.873520 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.873556 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:47:11.873666 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:47:11.873801 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2268,"thread_start_us":144,"threads_started":1}
I20260928 16:47:11.875216 13511 tablet_service.cc:1514] Processing DeleteTablet for tablet c01f6c289d5f485c8631865c1fa7ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:11 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:50208
I20260928 16:47:11.876140 13587 tablet_replica.cc:330] stopping tablet replica
I20260928 16:47:11.876462 13587 raft_consensus.cc:2229] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:11.876735 13587 raft_consensus.cc:2258] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:11.877738 13587 ts_tablet_manager.cc:1905] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:11.881569 13587 ts_tablet_manager.cc:1918] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:11.881709 13587 log.cc:1196] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/c01f6c289d5f485c8631865c1fa7ef76
I20260928 16:47:11.881997 13587 ts_tablet_manager.cc:1939] T c01f6c289d5f485c8631865c1fa7ef76 P 0e6495fa75eb4137a8b328c4744b17ee: Deleting consensus metadata
I20260928 16:47:11.883889 13374 catalog_manager.cc:4841] TS 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377): tablet c01f6c289d5f485c8631865c1fa7ef76 (table test-table [id=1a3b24a7b59e438ab99278df04a2b844]) successfully deleted
W20260928 16:47:11.964200 13582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:11.964501 13582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:11.964547 13582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:11.969105 13582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:11.969183 13582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:11.969247 13582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:11.974711 13582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:38559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=46213
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:11.975864 13582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:11.976847 13582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:47:11.987038 13582 server_base.cc:1034] running on GCE node
W20260928 16:47:11.986982 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.986982 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:11.986982 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:11.987807 13582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:11.988517 13582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:11.989763 13582 hybrid_clock.cc:648] HybridClock initialized: now 1790614031989735 us; error 41 us; skew 500 ppm
Sep 28 16:47:11 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:11.993270 13582 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:11.994302 13582 webserver.cc:460] Webserver started at http://127.30.196.2:46213/ using document root <none> and password file <none>
I20260928 16:47:11.994920 13582 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:11.995002 13582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:12.000728 13582 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:47:12.003541 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.005086 13582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:12.005210 13582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "4c74cd53a4414018912d79484cc08ac0"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "0c191fabb962b5f9975eb85c1da8ef8a"
server_key_iv: "fa321aa064325b88836b3e2efdd50fc1"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.005822 13582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:12.023293 13582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:12.026731 13582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:12.026892 13582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:12.027459 13582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:12.028649 13609 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 16:47:12.032778 13609 ts_tablet_manager.cc:1852] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 16:47:12.033203 13609 ts_tablet_manager.cc:1905] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:12.037499 13609 ts_tablet_manager.cc:1918] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:12.037672 13609 log.cc:1196] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/c01f6c289d5f485c8631865c1fa7ef76
I20260928 16:47:12.038072 13609 ts_tablet_manager.cc:1939] T c01f6c289d5f485c8631865c1fa7ef76 P 4c74cd53a4414018912d79484cc08ac0: Deleting consensus metadata
I20260928 16:47:12.038460 13582 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 16:47:12.038535 13582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260928 16:47:12.038666 13582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:12.038717 13582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.049816 13582 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38559
I20260928 16:47:12.049880 13716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38559 every 8 connection(s)
I20260928 16:47:12.050874 13582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:12.056809 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13582
I20260928 16:47:12.057288 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:38673
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=43899
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:12 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614031, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:12.068017 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.053121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:52849 (local address 127.30.196.62:45397)
0928 16:47:12.053317 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.053323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.054350 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 16:47:12.055364 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.055367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.055451 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.055548 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:12.057331 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.057979 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.059229 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.059484 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.059606 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.062851 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:12.062858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:12.062865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:12.062905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:12.065021 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.065767 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.065774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.065779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.065886 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.066377 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.066383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.066386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.066701 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:12.066822 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.067546 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.067757 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:47:12.068778 13719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.053562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:52849)
0928 16:47:12.054140 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.054188 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.055089 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.055846 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.055855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.056429 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:12.057143 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.057163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.058144 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.058152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.059068 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.059077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.059698 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.059703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.059891 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.060584 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:12.060603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:12.062611 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:12.065191 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:12.065197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:12.065213 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:12.065654 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:12.066055 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:12.066057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:12.066059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:12.066276 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:12.066940 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:12.066952 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:12.067266 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.067451 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.067816 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":144,"threads_started":1}
I20260928 16:47:12.070214 13717 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:12.070489 13717 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:12.071024 13717 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:12.072302 13389 ts_manager.cc:194] Registered new tserver with Master: 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
I20260928 16:47:12.073083 13389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:52849
W20260928 16:47:12.182243 13720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:12.182646 13720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:12.182773 13720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:12.187044 13720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:12.187182 13720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:12.187263 13720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:12.193424 13720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:38673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=43899
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:12.194518 13720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:12.195708 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:12.207098 13725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.207155 13726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.207098 13728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:12.207134 13720 server_base.cc:1034] running on GCE node
I20260928 16:47:12.208151 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:12.208878 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:12.210062 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790614032210028 us; error 99 us; skew 500 ppm
Sep 28 16:47:12 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:12.213131 13720 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:12.214071 13720 webserver.cc:460] Webserver started at http://127.30.196.3:43899/ using document root <none> and password file <none>
I20260928 16:47:12.214596 13720 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:12.214672 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:12.218569 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:12.221180 13735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.222635 13720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:47:12.222743 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "1bbd392678684d85ba071f9685e05568"
format_stamp: "Formatted at 2026-09-28 16:47:11 on dist-test-slave-5rg1"
server_key: "a19daaeb43cc2e686a20ddb6b567a0ba"
server_key_iv: "c781def01ab9675a26537735770b72f0"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.223277 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:12.252323 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:12.256453 13720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:12.256649 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:12.257284 13720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:12.258169 13720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:12.258222 13720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.258267 13720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:12.258284 13720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.269339 13720 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:38673
I20260928 16:47:12.269373 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:38673 every 8 connection(s)
I20260928 16:47:12.270746 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:12.275774 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13720
Sep 28 16:47:12 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:12.286978 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.276609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.30.196.62:45397)
0928 16:47:12.276861 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.276865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.276963 (+ 98us) server_negotiation.cc:407] Connection header received
0928 16:47:12.277076 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.277081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.277127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.277225 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:12.278148 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.278759 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.279889 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.280196 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.280343 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.282093 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:12.282097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:12.282101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:12.282145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:12.284557 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.285087 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.285091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.285093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.285190 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.285582 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.285586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.285587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.285978 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:12.286154 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.286491 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.286662 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260928 16:47:12.287932 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.273310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:48935 (local address 127.30.196.62:45397)
0928 16:47:12.273477 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.273482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.274550 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 16:47:12.276157 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.276161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.276225 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.276311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:12.277897 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.278434 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.279651 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.279950 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.280080 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.282887 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:12.282893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:12.282909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:12.282967 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:12.284946 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.285759 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.285762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.285764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.285841 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.286467 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.286470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.286472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.286685 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:12.286783 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.287575 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.287745 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":1920}
I20260928 16:47:12.289733 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.273702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:48935)
0928 16:47:12.274317 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.274363 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.275843 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.276555 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.276564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.277117 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:12.277729 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.277744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.278581 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.278585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.279508 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.279518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.280132 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.280137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.280325 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.281037 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:12.281055 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:12.282682 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:12.285134 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:12.285141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:12.285158 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:12.285631 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:12.286097 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:12.286101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:12.286103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:12.286350 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:12.286846 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:12.286854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:12.287239 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.288126 (+ 887us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.288602 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":220,"threads_started":1}
I20260928 16:47:12.291026 13851 heartbeater.cc:347] Connected to a master server at 127.30.196.62:45397
I20260928 16:47:12.291311 13851 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:12.291908 13851 heartbeater.cc:511] Master 127.30.196.62:45397 requested a full tablet report, sending...
I20260928 16:47:12.293203 13389 ts_manager.cc:194] Registered new tserver with Master: 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673)
I20260928 16:47:12.293798 13389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:48935
I20260928 16:47:12.301779 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:12.306816 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.302617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.30.196.62:45397)
0928 16:47:12.302787 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.302792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.302973 (+ 181us) server_negotiation.cc:407] Connection header received
0928 16:47:12.303217 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.303219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.303332 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.303452 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:12.303785 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.304380 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.305300 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.305521 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.305649 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.305980 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:12.306409 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:12.306554 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.306607 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.306664 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:47:12.307499 13389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:12.311101 13389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:12.319962 13511 tablet_service.cc:1467] Processing CreateTablet for tablet 28d11781a63546a6acc8973c79701de0 (DEFAULT_TABLE table=test-table [id=49e6758c052d4d8fb98e76f7fda3e156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:12.320647 13511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d11781a63546a6acc8973c79701de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:12.323909 13469 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap starting.
I20260928 16:47:12.324831 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.319745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.30.196.2:38559)
0928 16:47:12.320084 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.320096 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.320120 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:47:12.320205 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.320210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.320437 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.320572 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:12.320882 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.322092 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.323371 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.324231 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.324374 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.324535 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.324592 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.324680 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":137,"threads_started":1}
I20260928 16:47:12.324821 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.319216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.0.0.1:42306)
0928 16:47:12.319477 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.319491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.319610 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.320584 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.320587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.320607 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:12.320733 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.320741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.322268 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.322275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.323223 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.323235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.324377 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.324380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.324473 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.324496 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.324575 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.324656 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":119,"threads_started":1}
I20260928 16:47:12.325006 13469 tablet_bootstrap.cc:654] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:12.326515 13469 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: No bootstrap required, opened a new log
I20260928 16:47:12.326619 13469 ts_tablet_manager.cc:1397] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 16:47:12.327076 13469 raft_consensus.cc:348] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.326627 13651 tablet_service.cc:1467] Processing CreateTablet for tablet 28d11781a63546a6acc8973c79701de0 (DEFAULT_TABLE table=test-table [id=49e6758c052d4d8fb98e76f7fda3e156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:12.327207 13469 raft_consensus.cc:374] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:12.327247 13469 raft_consensus.cc:729] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6495fa75eb4137a8b328c4744b17ee, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.327387 13469 consensus_queue.cc:260] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.327898 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d11781a63546a6acc8973c79701de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:12.328306 13469 ts_tablet_manager.cc:1428] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:47:12.332856 13609 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Bootstrap starting.
W20260928 16:47:12.334012 13579 tablet.cc:2369] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:12.334159 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.318896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:38673 (local address 127.0.0.1:44956)
0928 16:47:12.319024 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.319041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.319132 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.324577 (+ 5445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.324581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.324602 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:12.324698 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.324704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.331745 (+ 7041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.331749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.332504 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.332515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.333709 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.333713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.333807 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.333829 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.333924 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.334018 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:47:12.334183 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.319115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.30.196.3:38673)
0928 16:47:12.324022 (+ 4907us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.324037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.324065 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:47:12.324151 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.324157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.324388 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.324867 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:12.325218 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.331551 (+ 6333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.332700 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.333572 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.333709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.333863 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.333952 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.334052 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4730,"thread_start_us":144,"threads_started":1}
I20260928 16:47:12.335410 13609 tablet_bootstrap.cc:654] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:12.336269 13609 log.cc:824] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:12.337150 13783 tablet_service.cc:1467] Processing CreateTablet for tablet 28d11781a63546a6acc8973c79701de0 (DEFAULT_TABLE table=test-table [id=49e6758c052d4d8fb98e76f7fda3e156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:12.338069 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d11781a63546a6acc8973c79701de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:12.338359 13609 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: No bootstrap required, opened a new log
I20260928 16:47:12.338552 13609 ts_tablet_manager.cc:1397] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 16:47:12.341868 13609 raft_consensus.cc:348] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.342136 13609 raft_consensus.cc:374] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:12.342198 13609 raft_consensus.cc:729] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c74cd53a4414018912d79484cc08ac0, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.342689 13609 consensus_queue.cc:260] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.343430 13717 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:12.343823 13609 ts_tablet_manager.cc:1428] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260928 16:47:12.345450 13866 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Bootstrap starting.
I20260928 16:47:12.347666 13866 tablet_bootstrap.cc:654] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:12.348790 13866 log.cc:824] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:12.350692 13866 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: No bootstrap required, opened a new log
I20260928 16:47:12.350860 13866 ts_tablet_manager.cc:1397] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:47:12.354066 13866 raft_consensus.cc:348] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.354303 13866 raft_consensus.cc:374] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:12.354336 13866 raft_consensus.cc:729] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbd392678684d85ba071f9685e05568, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.354704 13866 consensus_queue.cc:260] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.355386 13851 heartbeater.cc:503] Master 127.30.196.62:45397 was elected leader, sending a full tablet report...
I20260928 16:47:12.355720 13866 ts_tablet_manager.cc:1428] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:47:12.424120 13868 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:12.424314 13868 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.425419 13868 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559), 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377)
I20260928 16:47:12.430164 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.425753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.30.196.3:35949)
0928 16:47:12.425978 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.425996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.426119 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.426474 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.426477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.426499 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:12.426636 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.426646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.427829 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.427834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.428774 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.428784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.429673 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.429677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.429798 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.429824 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.429900 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.429987 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":43}
I20260928 16:47:12.430178 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.426014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:35949 (local address 127.30.196.2:38559)
0928 16:47:12.426188 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.426193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.426211 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:47:12.426269 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.426272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.426335 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.426459 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:12.426769 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.427641 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.428944 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.429525 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.429632 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.429683 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.429950 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.430036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:47:12.430353 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.425923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:49171 (local address 127.30.196.1:33377)
0928 16:47:12.426306 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.426311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.426581 (+ 270us) server_negotiation.cc:407] Connection header received
0928 16:47:12.426695 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.426699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.426761 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.426851 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:12.427208 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.427829 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.429197 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.429690 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.429809 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.429945 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.430133 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.430225 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260928 16:47:12.430445 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.425753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.30.196.3:49171)
0928 16:47:12.426499 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.426512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.426600 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.426891 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:12.426894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:12.426923 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:12.427057 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.427064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.428038 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.428041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.429053 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:12.429064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.429832 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.429835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:12.429982 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.430008 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:47:12.430093 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:47:12.430165 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":377,"thread_start_us":155,"threads_started":1}
I20260928 16:47:12.431071 13532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d11781a63546a6acc8973c79701de0" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6495fa75eb4137a8b328c4744b17ee" is_pre_election: true
I20260928 16:47:12.431288 13671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d11781a63546a6acc8973c79701de0" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cd53a4414018912d79484cc08ac0" is_pre_election: true
I20260928 16:47:12.431439 13532 raft_consensus.cc:2454] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 0.
I20260928 16:47:12.431551 13671 raft_consensus.cc:2454] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 0.
I20260928 16:47:12.431952 13738 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee, 1bbd392678684d85ba071f9685e05568; no voters:
I20260928 16:47:12.432240 13868 raft_consensus.cc:2781] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:12.432363 13868 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:12.432425 13868 raft_consensus.cc:3037] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:12.433530 13868 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.433959 13868 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 election: Requested vote from peers 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559), 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377)
I20260928 16:47:12.434290 13671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d11781a63546a6acc8973c79701de0" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c74cd53a4414018912d79484cc08ac0"
I20260928 16:47:12.434435 13671 raft_consensus.cc:3037] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:12.434477 13532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d11781a63546a6acc8973c79701de0" candidate_uuid: "1bbd392678684d85ba071f9685e05568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e6495fa75eb4137a8b328c4744b17ee"
I20260928 16:47:12.434650 13532 raft_consensus.cc:3037] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:12.435573 13671 raft_consensus.cc:2454] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 1.
I20260928 16:47:12.435853 13532 raft_consensus.cc:2454] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbd392678684d85ba071f9685e05568 in term 1.
I20260928 16:47:12.435906 13737 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0; no voters:
I20260928 16:47:12.436090 13868 raft_consensus.cc:2781] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:12.436333 13868 raft_consensus.cc:686] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbd392678684d85ba071f9685e05568, State: Running, Role: LEADER
I20260928 16:47:12.436699 13868 consensus_queue.cc:237] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.440277 13389 catalog_manager.cc:5495] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bbd392678684d85ba071f9685e05568 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "1bbd392678684d85ba071f9685e05568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } health_report { overall_health: UNKNOWN } } }
I20260928 16:47:12.454957 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.450746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.30.196.3:38673)
0928 16:47:12.450942 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.450947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.450964 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:47:12.451171 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.451175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.451236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.451363 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:12.451651 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.452248 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.453284 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.453533 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.453662 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.454018 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:12.454473 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:12.454619 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.454702 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.454766 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:47:12.461493 13671 raft_consensus.cc:1261] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Refusing update from remote peer 1bbd392678684d85ba071f9685e05568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:12.461493 13532 raft_consensus.cc:1261] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Refusing update from remote peer 1bbd392678684d85ba071f9685e05568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:12.462420 13871 consensus_queue.cc:1035] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:12.462677 13868 consensus_queue.cc:1035] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:12.469805 13874 mvcc.cc:205] Tried to move back new op lower bound from 7334355076952965120 to 7334355076863741952. Current Snapshot: MvccSnapshot[applied={T|T < 7334355076952965120}]
I20260928 16:47:12.478263 13388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:12.478468 13388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:12.480947 13388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Sending DeleteTablet for 3 replicas of tablet 28d11781a63546a6acc8973c79701de0
I20260928 16:47:12.481658 13511 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d11781a63546a6acc8973c79701de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:12 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:50208
I20260928 16:47:12.481813 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13427
I20260928 16:47:12.481797 13783 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d11781a63546a6acc8973c79701de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:12 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:44956
I20260928 16:47:12.482234 13651 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d11781a63546a6acc8973c79701de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:12 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:42306
I20260928 16:47:12.483436 13885 tablet_replica.cc:330] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: stopping tablet replica
I20260928 16:47:12.483840 13885 raft_consensus.cc:2229] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:12.484203 13885 raft_consensus.cc:2258] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:12.485865 13885 ts_tablet_manager.cc:1905] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:12.488446 13884 tablet_replica.cc:330] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: stopping tablet replica
I20260928 16:47:12.488817 13884 raft_consensus.cc:2229] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:12.489274 13884 raft_consensus.cc:2258] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:12.489437 13885 ts_tablet_manager.cc:1918] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:12.489526 13885 log.cc:1196] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/28d11781a63546a6acc8973c79701de0
I20260928 16:47:12.489816 13885 ts_tablet_manager.cc:1939] T 28d11781a63546a6acc8973c79701de0 P 4c74cd53a4414018912d79484cc08ac0: Deleting consensus metadata
W20260928 16:47:12.491074 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:33377 (error 108)
I20260928 16:47:12.491093 13884 ts_tablet_manager.cc:1905] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:12.491705 13373 catalog_manager.cc:4841] TS 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): tablet 28d11781a63546a6acc8973c79701de0 (table test-table [id=49e6758c052d4d8fb98e76f7fda3e156]) successfully deleted
W20260928 16:47:12.492776 13585 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:47:12.492045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33377 (local address 127.0.0.1:50218)
0928 16:47:12.492184 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.492202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.492294 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:12.492649 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:47:12.492937 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13582
I20260928 16:47:12.494602 13884 ts_tablet_manager.cc:1918] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:12.494697 13884 log.cc:1196] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/28d11781a63546a6acc8973c79701de0
I20260928 16:47:12.495028 13884 ts_tablet_manager.cc:1939] T 28d11781a63546a6acc8973c79701de0 P 1bbd392678684d85ba071f9685e05568: Deleting consensus metadata
W20260928 16:47:12.494370 13374 catalog_manager.cc:4584] TS 0e6495fa75eb4137a8b328c4744b17ee (127.30.196.1:33377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28d11781a63546a6acc8973c79701de0: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:47:12.496284 13373 catalog_manager.cc:4841] TS 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): tablet 28d11781a63546a6acc8973c79701de0 (table test-table [id=49e6758c052d4d8fb98e76f7fda3e156]) successfully deleted
I20260928 16:47:12.500381 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13720
I20260928 16:47:12.506733 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13355
I20260928 16:47:12.513722 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--webserver_interface=127.30.196.62
--webserver_port=41447
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:45397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:12.630744 13887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:12.631076 13887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:12.631119 13887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:12.635255 13887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:12.635326 13887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:12.635345 13887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:12.635362 13887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:12.635377 13887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:12.641232 13887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:45397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:45397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:12.642406 13887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:12.643468 13887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:12.652287 13893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:12.652360 13887 server_base.cc:1034] running on GCE node
W20260928 16:47:12.652509 13895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.652292 13892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:12.653096 13887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:12.654294 13887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:12.655474 13887 hybrid_clock.cc:648] HybridClock initialized: now 1790614032655456 us; error 34 us; skew 500 ppm
Sep 28 16:47:12 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:12.658576 13887 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:12.659602 13887 webserver.cc:460] Webserver started at http://127.30.196.62:41447/ using document root <none> and password file <none>
I20260928 16:47:12.660255 13887 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:12.660324 13887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:12.664378 13887 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:12.666734 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.668221 13887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:12.668367 13887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "65c54f2c027b439c9c0702d3d71c6023"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "7e92cdcc23d433c69f1af465bbadb9be"
server_key_iv: "c3eac68d86b59661eb7bb39d1ef29d2e"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.668985 13887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:12.697229 13887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:12.700547 13887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:12.700861 13887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:12.708122 13887 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:45397
I20260928 16:47:12.708415 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:45397 every 8 connection(s)
I20260928 16:47:12.709317 13887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:12.711585 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13887
I20260928 16:47:12.712057 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:33377
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=39231
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:12.713809 13956 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:12.715489 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap starting.
I20260928 16:47:12.733615 13956 log.cc:824] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:12.745383 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:12.745805 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023: Bootstrap complete.
I20260928 16:47:12.749912 13956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:12.750321 13956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.750787 13956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:12.750944 13956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:12.751036 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:12.751160 13956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:47:12.753113 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:12.753444 13956 leader_election.cc:302] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c54f2c027b439c9c0702d3d71c6023; no voters:
I20260928 16:47:12.753801 13956 leader_election.cc:288] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:47:12.753888 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:47:12.754099 13960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [term 3 LEADER]: Becoming Leader. State: Replica: 65c54f2c027b439c9c0702d3d71c6023, State: Running, Role: LEADER
I20260928 16:47:12.754396 13960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } }
I20260928 16:47:12.755051 13956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:12.755362 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c54f2c027b439c9c0702d3d71c6023. Latest consensus state: current_term: 3 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:12.755483 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:12.755873 13964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:12.756006 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "65c54f2c027b439c9c0702d3d71c6023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c54f2c027b439c9c0702d3d71c6023" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 45397 } } }
I20260928 16:47:12.756317 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c54f2c027b439c9c0702d3d71c6023 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:12.761217 13964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:12.762056 13964 catalog_manager.cc:1259] Loaded cluster ID: 3af27c6dad6b43fc9ba11349ccb66924
I20260928 16:47:12.762151 13964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 16:47:12.762965 13964 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f061ef099ad google::LogMessage::Fail() at ??:0
@ 0x7f061ef0c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f061ef094de google::LogMessage::Flush() at ??:0
@ 0x7f061ef0aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f062984cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f062984dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f062984a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f062988e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f062115b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f062115bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f062115d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f062114c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f06227906db start_thread at ??:0
@ 0x7f061da0761f clone at ??:0
W20260928 16:47:12.843775 13958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:12.844123 13958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:12.844234 13958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:12.848471 13958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:12.848560 13958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:12.848670 13958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:12.854218 13958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:33377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=39231
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:12.855398 13958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:12.856536 13958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:12.867030 13997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.867110 13998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.867033 14000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:12.867368 13958 server_base.cc:1034] running on GCE node
I20260928 16:47:12.868949 13958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:12.870328 13958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:12.871524 13958 hybrid_clock.cc:648] HybridClock initialized: now 1790614032871506 us; error 38 us; skew 500 ppm
Sep 28 16:47:12 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:12.874560 13958 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:12.875551 13958 webserver.cc:460] Webserver started at http://127.30.196.1:39231/ using document root <none> and password file <none>
I20260928 16:47:12.876178 13958 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:12.876245 13958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:12.880270 13958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:12.882457 14008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.883836 13958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:47:12.883961 13958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "0e6495fa75eb4137a8b328c4744b17ee"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "af3d8076be624a1a4c924b118f2e8327"
server_key_iv: "d4dbf95808421d2bed73778dab29f0d9"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.884509 13958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:12.897303 13958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:12.900665 13958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:12.900873 13958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:12.901501 13958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:12.902644 14015 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:47:12.908388 13958 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:47:12.908484 13958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 16:47:12.908569 13958 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:47:12.910409 13958 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:47:12.910482 13958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:47:12.910768 14015 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap starting.
I20260928 16:47:12.921285 13958 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:33377
I20260928 16:47:12.921548 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:33377 every 8 connection(s)
I20260928 16:47:12.922343 13958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:12.929437 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 13958
I20260928 16:47:12.929980 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:38559
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=46213
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:12.933476 14015 log.cc:824] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:12.943427 14015 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:12.943985 14015 tablet_bootstrap.cc:492] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Bootstrap complete.
I20260928 16:47:12.945674 14015 ts_tablet_manager.cc:1397] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.011s
I20260928 16:47:12.949146 14015 raft_consensus.cc:348] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.949746 14015 raft_consensus.cc:729] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6495fa75eb4137a8b328c4744b17ee, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.950307 14015 consensus_queue.cc:260] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:12.951642 14015 ts_tablet_manager.cc:1428] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260928 16:47:13.006502 14125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:47:12.924451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:54267)
0928 16:47:12.924952 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:12.925009 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:12.925849 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:13.005436 (+ 79587us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":158,"threads_started":1}
W20260928 16:47:13.007227 14123 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:45397 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:45397: Client connection negotiation failed: client connection to 127.30.196.62:45397: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:47:13.008224 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.007835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:41311)
0928 16:47:13.008073 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.008153 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 16:47:13.009032 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.008779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:53455)
0928 16:47:13.008919 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.008969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 16:47:13.009280 14123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:47:13.051558 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:13.051858 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:13.051957 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:13.056077 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:13.056143 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:13.056234 14126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:13.061535 14126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:38559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=46213
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:13.062662 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:13.063760 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:13.073566 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:13.073832 14126 server_base.cc:1034] running on GCE node
W20260928 16:47:13.074008 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.074026 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:13.074484 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:13.075160 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:13.076395 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790614033076346 us; error 49 us; skew 500 ppm
Sep 28 16:47:13 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:13.079278 14126 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:13.080284 14126 webserver.cc:460] Webserver started at http://127.30.196.2:46213/ using document root <none> and password file <none>
I20260928 16:47:13.080879 14126 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:13.080979 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:13.084864 14126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 16:47:13.087075 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.088515 14126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:13.088685 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "4c74cd53a4414018912d79484cc08ac0"
format_stamp: "Formatted at 2026-09-28 16:47:10 on dist-test-slave-5rg1"
server_key: "0c191fabb962b5f9975eb85c1da8ef8a"
server_key_iv: "fa321aa064325b88836b3e2efdd50fc1"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.089265 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:13.105130 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:13.108361 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:13.108613 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:13.109362 14126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:13.110328 14126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:13.110407 14126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.110486 14126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:13.110534 14126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.120618 14126 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38559
I20260928 16:47:13.120639 14257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38559 every 8 connection(s)
I20260928 16:47:13.121737 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:13.126080 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.124303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:43471)
0928 16:47:13.124807 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.125326 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":120,"threads_started":1}
W20260928 16:47:13.126624 14258 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:45397 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:45397: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
I20260928 16:47:13.127319 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.127083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:34809)
0928 16:47:13.127207 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.127259 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:47:13.128199 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.127888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:46727)
0928 16:47:13.128083 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.128137 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 16:47:13.128484 14258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:47:13.129127 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14126
I20260928 16:47:13.129518 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:38673
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:45397
--webserver_port=43899
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:13.244261 14261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:13.244562 14261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:13.244609 14261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:13.249084 14261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:13.249164 14261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:13.249229 14261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:13.254792 14261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:38673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=43899
--tserver_master_addrs=127.30.196.62:45397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:13.256181 14261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:13.257300 14261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:13.267920 14267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.267928 14269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.268046 14266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:13.268141 14261 server_base.cc:1034] running on GCE node
I20260928 16:47:13.268750 14261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:13.269496 14261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:13.270715 14261 hybrid_clock.cc:648] HybridClock initialized: now 1790614033270691 us; error 38 us; skew 500 ppm
Sep 28 16:47:13 dist-test-slave-5rg1 krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:13.273710 14261 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:13.274629 14261 webserver.cc:460] Webserver started at http://127.30.196.3:43899/ using document root <none> and password file <none>
I20260928 16:47:13.275168 14261 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:13.275228 14261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:13.279819 14261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:13.282426 14276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.284078 14261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:13.284242 14261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "1bbd392678684d85ba071f9685e05568"
format_stamp: "Formatted at 2026-09-28 16:47:11 on dist-test-slave-5rg1"
server_key: "a19daaeb43cc2e686a20ddb6b567a0ba"
server_key_iv: "c781def01ab9675a26537735770b72f0"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.284845 14261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:13.323828 14261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:13.327167 14261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:13.327391 14261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:13.328068 14261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:13.328958 14261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:13.329033 14261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.329105 14261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:13.329150 14261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.338691 14261 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:38673
I20260928 16:47:13.338722 14390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:38673 every 8 connection(s)
I20260928 16:47:13.339736 14261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:13.343789 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.342050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:55969)
0928 16:47:13.342532 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.342955 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
W20260928 16:47:13.344348 14391 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:45397 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:45397: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
I20260928 16:47:13.345165 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.344914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:56535)
0928 16:47:13.345063 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.345112 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 16:47:13.345799 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.345615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:37043)
0928 16:47:13.345716 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.345756 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:47:13.346156 14391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:47:13.347108 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14261
I20260928 16:47:14.010581 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.010038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:33105)
0928 16:47:14.010348 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.010454 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260928 16:47:14.129555 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.129075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:36523)
0928 16:47:14.129359 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.129431 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260928 16:47:14.254061 14397 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:14.254315 14397 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:14.255582 14397 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
Sep 28 16:47:14 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.2@KRBTEST.COM
Sep 28 16:47:14 dist-test-slave-5rg1 krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.3@KRBTEST.COM
I20260928 16:47:14.272518 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.256418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38559 (local address 127.30.196.1:39209)
0928 16:47:14.256745 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.256760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.256858 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.258759 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.258766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.259494 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:14.260166 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.260185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.261596 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.261601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.262723 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.262736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.263549 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.263552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.263782 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.264471 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:14.264487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:14.266320 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:14.269144 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.269154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.269173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.269661 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.270167 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.270170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.270172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.270301 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.271188 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:14.271195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:14.271612 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.271875 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.272284 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":162,"spinlock_wait_cycles":89216,"thread_start_us":74,"threads_started":1}
I20260928 16:47:14.272518 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.256109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:38673 (local address 127.30.196.1:55323)
0928 16:47:14.256269 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.256291 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.256445 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.258759 (+ 2314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.258766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.259548 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:14.260166 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.260185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.261686 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.261689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.262723 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.262736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.263779 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.263782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.263848 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.264606 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:14.264618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:14.266705 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:14.269254 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.269257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.269260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.269645 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.270092 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.270096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.270097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.270302 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.271188 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:14.271195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:14.271612 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.271875 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.272337 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":437,"spinlock_wait_cycles":4480}
I20260928 16:47:14.272598 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.256653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:39209 (local address 127.30.196.2:38559)
0928 16:47:14.257005 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.257027 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.257060 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:47:14.257827 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.257834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.258284 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.258633 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:14.260381 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.261408 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.262873 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.263387 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.263532 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.266656 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.266664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.266682 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.266722 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.268997 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.269807 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.269815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.269821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.269992 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.270440 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.270444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.270446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.271081 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.271194 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.272105 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.272404 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":135,"threads_started":1}
I20260928 16:47:14.273677 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.256361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:55323 (local address 127.30.196.3:38673)
0928 16:47:14.256792 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.256804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.256872 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:47:14.257798 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.257808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.258284 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.258633 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:14.260391 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.261519 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.262870 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.263619 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.263813 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.266899 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.266903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.266914 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.266944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.269105 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.269775 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.269781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.269792 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.269974 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.270440 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.270444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.270446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.271030 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.271183 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.272075 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.272414 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":149,"threads_started":1}
W20260928 16:47:14.275316 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
W20260928 16:47:14.275599 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
I20260928 16:47:14.275751 14010 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee; no voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0
I20260928 16:47:14.276062 14397 raft_consensus.cc:2726] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:47:14.347451 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.346771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:46535)
0928 16:47:14.347212 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.347276 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":67,"threads_started":1}
I20260928 16:47:15.012247 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:15.011634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:60261)
0928 16:47:15.012056 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:15.012124 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":91,"threads_started":1}
I20260928 16:47:15.131062 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:15.130531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:35131)
0928 16:47:15.130831 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:15.130962 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260928 16:47:15.348721 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:15.348271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:35835)
0928 16:47:15.348531 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:15.348596 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260928 16:47:16.013536 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:16.013109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:39469)
0928 16:47:16.013367 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:16.013430 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260928 16:47:16.019533 14406 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:16.019711 14406 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:16.020162 14406 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
W20260928 16:47:16.020984 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
W20260928 16:47:16.021128 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
I20260928 16:47:16.021188 14010 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee; no voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0
I20260928 16:47:16.021386 14406 raft_consensus.cc:2726] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:47:16.132364 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:16.131905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:45611)
0928 16:47:16.132203 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:16.132264 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
I20260928 16:47:16.350242 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:16.349687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:36757)
0928 16:47:16.350040 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:16.350114 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260928 16:47:17.014763 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:17.014336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:37987)
0928 16:47:17.014598 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:17.014660 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260928 16:47:17.133626 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:17.133173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:59799)
0928 16:47:17.133452 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:17.133523 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
I20260928 16:47:17.351531 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:17.351065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:50559)
0928 16:47:17.351371 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:17.351433 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260928 16:47:18.016489 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:18.015943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:60989)
0928 16:47:18.016315 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:18.016384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":90,"threads_started":1}
I20260928 16:47:18.135267 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:18.134681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:55009)
0928 16:47:18.135090 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:18.135161 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":113,"threads_started":1}
I20260928 16:47:18.353199 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:18.352567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:48079)
0928 16:47:18.352976 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:18.353064 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260928 16:47:18.606076 14415 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:18.606225 14415 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:18.606746 14415 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
W20260928 16:47:18.607723 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
W20260928 16:47:18.607883 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
I20260928 16:47:18.607945 14010 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee; no voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0
I20260928 16:47:18.608160 14415 raft_consensus.cc:2726] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:47:19.017966 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:19.017424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:57249)
0928 16:47:19.017732 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:19.017797 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260928 16:47:19.136953 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:19.136415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:36745)
0928 16:47:19.136709 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:19.136776 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260928 16:47:19.354545 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:19.354082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:40855)
0928 16:47:19.354371 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:19.354436 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260928 16:47:20.019367 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:20.018921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:58895)
0928 16:47:20.019201 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:20.019258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260928 16:47:20.138515 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:20.137965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:42939)
0928 16:47:20.138345 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:20.138412 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260928 16:47:20.356076 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:20.355416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:38513)
0928 16:47:20.355868 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:20.355933 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":83,"threads_started":1}
I20260928 16:47:21.020984 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:21.020466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:57349)
0928 16:47:21.020762 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:21.020827 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260928 16:47:21.139809 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:21.139329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:52779)
0928 16:47:21.139592 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:21.139649 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260928 16:47:21.357399 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:21.356909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:42945)
0928 16:47:21.357241 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:21.357304 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260928 16:47:22.022396 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:22.021910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:53113)
0928 16:47:22.022229 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:22.022301 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20260928 16:47:22.141089 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:22.140642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:50529)
0928 16:47:22.140898 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:22.140957 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
I20260928 16:47:22.358824 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:22.358318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:57897)
0928 16:47:22.358657 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:22.358722 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20260928 16:47:23.024071 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:23.023502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:34677)
0928 16:47:23.023852 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:23.023970 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":78,"threads_started":1}
I20260928 16:47:23.142490 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:23.141968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:42055)
0928 16:47:23.142293 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:23.142363 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260928 16:47:23.360255 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:23.359831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:49961)
0928 16:47:23.360102 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:23.360158 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260928 16:47:23.428012 14432 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:23.428144 14432 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:23.428510 14432 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
W20260928 16:47:23.429440 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
W20260928 16:47:23.429631 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
I20260928 16:47:23.429736 14010 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee; no voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0
I20260928 16:47:23.429929 14432 raft_consensus.cc:2726] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:47:24.025338 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:24.024897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:56199)
0928 16:47:24.025172 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:24.025240 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260928 16:47:24.144178 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:24.143431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:48223)
0928 16:47:24.143860 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:24.144045 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260928 16:47:24.361606 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:24.361116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:33431)
0928 16:47:24.361430 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:24.361499 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260928 16:47:25.026811 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:25.026343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:39521)
0928 16:47:25.026644 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:25.026710 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260928 16:47:25.145501 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:25.145044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:60211)
0928 16:47:25.145333 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:25.145398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260928 16:47:25.362974 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:25.362542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:40967)
0928 16:47:25.362816 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:25.362875 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260928 16:47:26.028200 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:26.027755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:56421)
0928 16:47:26.028035 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:26.028100 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260928 16:47:26.146755 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:26.146307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:47965)
0928 16:47:26.146597 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:26.146656 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260928 16:47:26.364267 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:26.363843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:53411)
0928 16:47:26.364118 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:26.364171 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260928 16:47:27.029570 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:27.029120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:54937)
0928 16:47:27.029394 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:27.029459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260928 16:47:27.148037 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:27.147510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:50045)
0928 16:47:27.147861 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:27.147933 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260928 16:47:27.365622 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:27.365135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.3:54505)
0928 16:47:27.365428 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:27.365516 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260928 16:47:27.407433 14445 raft_consensus.cc:482] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:27.407598 14445 raft_consensus.cc:504] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbd392678684d85ba071f9685e05568" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 38673 } } peers { permanent_uuid: "4c74cd53a4414018912d79484cc08ac0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38559 } } peers { permanent_uuid: "0e6495fa75eb4137a8b328c4744b17ee" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33377 } }
I20260928 16:47:27.408177 14445 leader_election.cc:288] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673), 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559)
W20260928 16:47:27.409029 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c74cd53a4414018912d79484cc08ac0 (127.30.196.2:38559): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
W20260928 16:47:27.409161 14010 leader_election.cc:341] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bbd392678684d85ba071f9685e05568 (127.30.196.3:38673): Not found: Tablet not found: 28d11781a63546a6acc8973c79701de0
I20260928 16:47:27.409245 14010 leader_election.cc:302] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e6495fa75eb4137a8b328c4744b17ee; no voters: 1bbd392678684d85ba071f9685e05568, 4c74cd53a4414018912d79484cc08ac0
I20260928 16:47:27.409457 14445 raft_consensus.cc:2726] T 28d11781a63546a6acc8973c79701de0 P 0e6495fa75eb4137a8b328c4744b17ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:47:28.031102 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:28.030646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.1:40989)
0928 16:47:28.030940 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:28.031002 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260928 16:47:28.149682 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:28.149236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:45397 (local address 127.30.196.2:52033)
0928 16:47:28.149512 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:28.149585 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:45397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260928 16:47:28.349031 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 13958
I20260928 16:47:28.355372 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14126
I20260928 16:47:28.361514 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14261
2026-09-28T16:47:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19374 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:47:28 dist-test-slave-5rg1 krb5kdc[14450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:47:28 dist-test-slave-5rg1 krb5kdc[14450](info): set up 2 sockets
Sep 28 16:47:28 dist-test-slave-5rg1 krb5kdc[14450](info): commencing operation
krb5kdc: starting...
W20260928 16:47:29.416162 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:47:29 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:47:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:47:29.550613 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:29.663162 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:29.663429 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:29.663470 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:29.667455 14463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:47:29.667514 14463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:47:29.667529 14463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:47:29.667544 14463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:29.667562 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:29.667580 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:29.667596 14463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:29.667611 14463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:29.673611 14463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:29.674630 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:29.675616 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:29.684362 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:29.684327 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:29.684332 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:29.684369 14463 server_base.cc:1034] running on GCE node
I20260928 16:47:29.685175 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:29.686334 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:29.687536 14463 hybrid_clock.cc:648] HybridClock initialized: now 1790614049687515 us; error 41 us; skew 500 ppm
Sep 28 16:47:29 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614049, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:29.690354 14463 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:29.691287 14463 webserver.cc:460] Webserver started at http://127.30.196.62:46611/ using document root <none> and password file <none>
I20260928 16:47:29.691928 14463 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:29.692049 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:29.692287 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:29.694309 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "9a951d9337fd40a187cee08016e1b46c"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "617a5c7d5f58c2df2f4233f996d532a8"
server_key_iv: "71a39864a19b745858bb82a25c26521d"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.694906 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "9a951d9337fd40a187cee08016e1b46c"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "617a5c7d5f58c2df2f4233f996d532a8"
server_key_iv: "71a39864a19b745858bb82a25c26521d"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.698597 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:47:29.701121 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:29.702358 14463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:47:29.702513 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "9a951d9337fd40a187cee08016e1b46c"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "617a5c7d5f58c2df2f4233f996d532a8"
server_key_iv: "71a39864a19b745858bb82a25c26521d"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.702632 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:29.719173 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:29.722707 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:29.722922 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:29.730571 14463 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36217
I20260928 16:47:29.730897 14530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36217 every 8 connection(s)
I20260928 16:47:29.731599 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:29.734397 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:29.737610 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14463
I20260928 16:47:29.737746 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:47:29.738057 31504 external_mini_cluster.cc:1428] Setting key 4b5076577572e8f5056819d3bcff1882
I20260928 16:47:29.740851 14531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap starting.
I20260928 16:47:29.743175 14531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:29.744017 14531 log.cc:824] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:29.745986 14531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: No bootstrap required, opened a new log
Sep 28 16:47:29 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:29.749485 14531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:29.749755 14531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:29.749840 14531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Initialized, Role: FOLLOWER
I20260928 16:47:29.750344 14531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:29.750504 14531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:29.750594 14531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:29.750711 14531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:29.751718 14531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:29.752107 14531 leader_election.cc:302] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a951d9337fd40a187cee08016e1b46c; no voters:
I20260928 16:47:29.752486 14531 leader_election.cc:288] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:47:29.752597 14536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:29.752840 14536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 LEADER]: Becoming Leader. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Running, Role: LEADER
I20260928 16:47:29.753221 14536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:29.753641 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:29.739562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.30.196.62:36217)
0928 16:47:29.740049 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:29.740065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:29.740101 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:47:29.740746 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:29.740764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:29.741178 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:29.741535 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:29.742386 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:29.743447 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:29.744460 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:29.744944 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:29.745125 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:29.747731 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:29.747739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:29.747761 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:29.747790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:29.750240 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:29.750783 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:29.750812 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:29.750821 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:29.750955 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:29.751275 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:29.751280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:29.751282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:29.751787 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:29.752002 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:29.752271 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:47:29.752602 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":132,"threads_started":1}
I20260928 16:47:29.753890 14531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:29.755017 14537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a951d9337fd40a187cee08016e1b46c. Latest consensus state: current_term: 1 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:47:29.755156 14537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:29.755643 14538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:47:29.755774 14538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:29.757058 14545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:29.759833 14545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:29.765403 14545 catalog_manager.cc:1347] Generated new cluster ID: 4150fffc4d5348af851196469cba638d
I20260928 16:47:29.765486 14545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:29.777109 14545 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:47:29.778420 14545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:29.789219 14545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Generated new TSK 0
I20260928 16:47:29.790132 14545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:47:29.827981 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:29.946808 14557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:29.947165 14557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:29.947252 14557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:29.951515 14557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:29.951628 14557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:29.951748 14557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:29.957826 14557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:29.959468 14557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:29.960606 14557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:29.970701 14562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:29.970727 14563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:29.970738 14557 server_base.cc:1034] running on GCE node
W20260928 16:47:29.970701 14565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:29.971387 14557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:29.972069 14557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:29.973232 14557 hybrid_clock.cc:648] HybridClock initialized: now 1790614049973212 us; error 32 us; skew 500 ppm
Sep 28 16:47:29 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614049, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:29.976025 14557 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:29.976912 14557 webserver.cc:460] Webserver started at http://127.30.196.1:35069/ using document root <none> and password file <none>
I20260928 16:47:29.977499 14557 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:29.977561 14557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:29.977797 14557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:29.979990 14557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "a27562ccdc1442d68da1ad4688ffb596"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "173fe8044e3c4ad72734219ed84e1076"
server_key_iv: "991e4cc2362078487e87105b25c1d0cd"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.980603 14557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "a27562ccdc1442d68da1ad4688ffb596"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "173fe8044e3c4ad72734219ed84e1076"
server_key_iv: "991e4cc2362078487e87105b25c1d0cd"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.984543 14557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:47:29.987056 14572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:29.988379 14557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:47:29.988534 14557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "a27562ccdc1442d68da1ad4688ffb596"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "173fe8044e3c4ad72734219ed84e1076"
server_key_iv: "991e4cc2362078487e87105b25c1d0cd"
server_key_version: "encryptionkey@0"
I20260928 16:47:29.988663 14557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:30.001945 14557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:30.005618 14557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:30.005898 14557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:30.006541 14557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:30.007424 14557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:30.007505 14557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.007586 14557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:30.007640 14557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.017351 14557 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45835
I20260928 16:47:30.017369 14687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45835 every 8 connection(s)
I20260928 16:47:30.018342 14557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:30.025655 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14557
I20260928 16:47:30.025797 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:47:30.026163 31504 external_mini_cluster.cc:1428] Setting key 3d15c22e641660fd0d1e0bb4f2643a5c
Sep 28 16:47:30 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614049, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:30.033174 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.020397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:33771 (local address 127.30.196.62:36217)
0928 16:47:30.020573 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.020578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.021424 (+ 846us) server_negotiation.cc:407] Connection header received
0928 16:47:30.022720 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.022724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.022774 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.022850 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:30.024144 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.024706 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.026160 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.026428 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.026518 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.029455 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:30.029458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:30.029460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:30.029489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:30.031000 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.031534 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.031538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.031540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.031627 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.032052 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.032057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.032059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.032250 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:30.032358 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.032868 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.033003 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:47:30.034058 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.020787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:33771)
0928 16:47:30.021267 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.021312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.022521 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.023026 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.023033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.023407 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:30.024005 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.024019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.024821 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.024823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.026032 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.026045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.026610 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.026622 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.026802 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.027505 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:30.027531 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:30.029271 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:30.031128 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.031134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.031151 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.031427 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.031792 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.031795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.031797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.031948 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.032358 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:30.032366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:30.032637 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.032830 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.033191 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260928 16:47:30.035542 14688 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:47:30.035809 14688 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:30.036367 14688 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:30.038002 14495 ts_manager.cc:194] Registered new tserver with Master: a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:47:30.039345 14495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:33771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:47:30.056159 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:30.175480 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:30.175743 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:30.175824 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:30.179863 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:30.179919 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:30.180032 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:30.185387 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:30.186503 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:30.187551 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:30.197913 14699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:30.197835 14700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:30.197835 14702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:30.197947 14693 server_base.cc:1034] running on GCE node
I20260928 16:47:30.198534 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:30.199235 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:30.200420 14693 hybrid_clock.cc:648] HybridClock initialized: now 1790614050200403 us; error 41 us; skew 500 ppm
Sep 28 16:47:30 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614050, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:30.203279 14693 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:30.204205 14693 webserver.cc:460] Webserver started at http://127.30.196.2:37977/ using document root <none> and password file <none>
I20260928 16:47:30.204770 14693 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:30.204861 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:30.205096 14693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:30.207195 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "39c5f1cb96ab4c67b7947f32918a0f2d"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fc1972c1b7e197273a354f1167749e78"
server_key_iv: "c2a874915fb0b1635fb148b2b57a8712"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.207824 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "39c5f1cb96ab4c67b7947f32918a0f2d"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fc1972c1b7e197273a354f1167749e78"
server_key_iv: "c2a874915fb0b1635fb148b2b57a8712"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.211463 14693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:30.213958 14709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.215261 14693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:30.215409 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "39c5f1cb96ab4c67b7947f32918a0f2d"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fc1972c1b7e197273a354f1167749e78"
server_key_iv: "c2a874915fb0b1635fb148b2b57a8712"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.215533 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:30.233784 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:30.237454 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:30.237684 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:30.238318 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:30.239194 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:30.239269 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.239342 14693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:30.239387 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.249228 14693 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:44131
I20260928 16:47:30.249246 14823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:44131 every 8 connection(s)
I20260928 16:47:30.250177 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:30.253816 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14693
I20260928 16:47:30.253981 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:47:30.254277 31504 external_mini_cluster.cc:1428] Setting key d63358eb9dcbbd0d101f653b4d5eb452
Sep 28 16:47:30 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614050, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:47:30.266280 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.252109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:56467 (local address 127.30.196.62:36217)
0928 16:47:30.252283 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.252287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.253181 (+ 894us) server_negotiation.cc:407] Connection header received
0928 16:47:30.254958 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.254962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.255022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.255117 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:30.257379 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.258036 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.259207 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.259445 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.259566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.262464 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:30.262467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:30.262470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:30.262505 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:30.264071 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.264622 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.264626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.264628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.264716 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.265067 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.265072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.265073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.265269 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:30.265392 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.265952 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.266099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:47:30.267241 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.252547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:56467)
0928 16:47:30.253015 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.253070 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.254720 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.255918 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.255926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.256454 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:30.257202 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.257225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.258173 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.258176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.259073 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.259083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.259608 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.259612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.259844 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.260469 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:30.260492 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:30.262271 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:30.264199 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.264205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.264221 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.264513 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.264811 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.264815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.264817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.264965 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.265382 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:30.265389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:30.265706 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.265909 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.266255 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":146,"threads_started":1}
I20260928 16:47:30.268674 14824 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
I20260928 16:47:30.268972 14824 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:30.269632 14824 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:30.270921 14495 ts_manager.cc:194] Registered new tserver with Master: 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
I20260928 16:47:30.271572 14495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:56467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:47:30.287256 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:30.403738 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:30.404076 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:30.404155 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:30.408504 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:30.408605 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:30.408700 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:30.414397 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:30.415566 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:30.416712 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:30.426887 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:30.426887 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:30.427060 14829 server_base.cc:1034] running on GCE node
W20260928 16:47:30.426998 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:30.427599 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:30.428277 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:30.429474 14829 hybrid_clock.cc:648] HybridClock initialized: now 1790614050429462 us; error 25 us; skew 500 ppm
Sep 28 16:47:30 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614050, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:30.432269 14829 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:30.433113 14829 webserver.cc:460] Webserver started at http://127.30.196.3:33349/ using document root <none> and password file <none>
I20260928 16:47:30.433640 14829 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:30.433704 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:30.433943 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:30.436167 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "eb37a9484cc3488dabb1f056f9c06897"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fe536b2f4841ef00b286cd1d33e182d8"
server_key_iv: "cb453b4d937e009ddfe4553d4bbf79e4"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.436702 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "eb37a9484cc3488dabb1f056f9c06897"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fe536b2f4841ef00b286cd1d33e182d8"
server_key_iv: "cb453b4d937e009ddfe4553d4bbf79e4"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.440591 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:30.443114 14844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.444808 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:30.444972 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "eb37a9484cc3488dabb1f056f9c06897"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fe536b2f4841ef00b286cd1d33e182d8"
server_key_iv: "cb453b4d937e009ddfe4553d4bbf79e4"
server_key_version: "encryptionkey@0"
I20260928 16:47:30.445101 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:30.460062 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:30.463327 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:30.463553 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:30.464213 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:30.465075 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:30.465121 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.465160 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:30.465175 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:30.474306 14829 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:37195
I20260928 16:47:30.474324 14958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:37195 every 8 connection(s)
I20260928 16:47:30.475244 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:30.484917 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 14829
I20260928 16:47:30.485098 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:47:30.485424 31504 external_mini_cluster.cc:1428] Setting key d4794105626bc52a98ace73719cba8f2
Sep 28 16:47:30 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614050, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:30.489835 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.477224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:56563 (local address 127.30.196.62:36217)
0928 16:47:30.477391 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.477396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.478321 (+ 925us) server_negotiation.cc:407] Connection header received
0928 16:47:30.479843 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.479846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.479923 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.480028 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:30.481230 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.481908 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.483091 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.483325 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.483441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.486234 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:30.486238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:30.486240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:30.486264 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:30.487918 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.488444 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.488447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.488448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.488525 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.488803 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.488805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.488806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.489044 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:30.489123 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.489616 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.489716 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:47:30.490741 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.477636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:56563)
0928 16:47:30.478106 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.478151 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.479565 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.480211 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.480219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.480599 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:30.481084 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.481098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.482034 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.482038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.482961 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.482971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.483488 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.483492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.483748 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.484346 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:30.484362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:30.486050 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:30.488057 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.488063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.488076 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.488348 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.488588 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:30.488590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:30.488591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:30.488722 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:30.489144 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:30.489149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:30.489397 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.489568 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.489841 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260928 16:47:30.492112 14959 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
I20260928 16:47:30.492386 14959 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:30.492923 14959 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:30.493963 14495 ts_manager.cc:194] Registered new tserver with Master: eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195)
I20260928 16:47:30.494643 14495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:56563
I20260928 16:47:30.499598 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:30.509648 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.501844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.30.196.62:36217)
0928 16:47:30.502073 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.502077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.502306 (+ 229us) server_negotiation.cc:407] Connection header received
0928 16:47:30.502392 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.502395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.502438 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.502517 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:30.503289 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.503909 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.504706 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.504932 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.505049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.506491 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:30.506494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:30.506497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:30.506519 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:30.508018 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.508391 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.508393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.508394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.508469 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:30.508767 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:30.508769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:30.508771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:30.509037 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:30.509181 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.509417 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.509523 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:47:30.512725 14495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:30.516503 14495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:30.539896 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.533485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.30.196.1:45835)
0928 16:47:30.534362 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.534381 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.534409 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:47:30.534494 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.534502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.534707 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.534873 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:30.535394 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.536671 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.538114 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539039 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539168 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539348 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.539544 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.539696 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":72,"threads_started":1}
I20260928 16:47:30.540012 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.533180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37195 (local address 127.0.0.1:45462)
0928 16:47:30.533796 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.533847 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.533975 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.535016 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.535020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.535038 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:30.535208 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.535213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.536379 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.536382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.538137 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.538150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539426 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.539629 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539652 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.539749 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.539827 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":168,"spinlock_wait_cycles":3328,"thread_start_us":92,"threads_started":1}
I20260928 16:47:30.540041 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.534272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44131 (local address 127.0.0.1:46082)
0928 16:47:30.534696 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.534709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.534798 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.535635 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.535638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.535658 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:30.535796 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.535802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.537262 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.537265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.538470 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.538479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539425 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.539598 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539626 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.539781 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.539857 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":32,"thread_start_us":113,"threads_started":1}
I20260928 16:47:30.540443 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.533296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.30.196.3:37195)
0928 16:47:30.534300 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.534316 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.534342 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:47:30.534447 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.534453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.534672 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.534844 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:30.535322 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.536249 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.538282 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539141 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539356 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.540200 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.540299 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":92,"threads_started":1}
I20260928 16:47:30.540573 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.533300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.0.0.1:47954)
0928 16:47:30.533893 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.533909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.534007 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.535046 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.535050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.535071 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:30.535257 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.535264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.536813 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.536816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.537907 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.537917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539183 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.539330 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539370 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.539484 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.540445 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260928 16:47:30.540654 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.534713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.30.196.2:44131)
0928 16:47:30.535156 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.535169 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.535191 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:30.535270 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.535275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.535471 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.535643 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:30.535918 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.537126 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.538590 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.539203 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.539313 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.539460 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.540381 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.540500 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":130,"threads_started":1}
I20260928 16:47:30.543380 14621 tablet_service.cc:1467] Processing CreateTablet for tablet db090b71570e49f39664e2f2c007137d (DEFAULT_TABLE table=test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:30.543453 14757 tablet_service.cc:1467] Processing CreateTablet for tablet db090b71570e49f39664e2f2c007137d (DEFAULT_TABLE table=test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:30.543450 14892 tablet_service.cc:1467] Processing CreateTablet for tablet db090b71570e49f39664e2f2c007137d (DEFAULT_TABLE table=test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:30.544603 14757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db090b71570e49f39664e2f2c007137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:30.544620 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db090b71570e49f39664e2f2c007137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:30.544970 14892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db090b71570e49f39664e2f2c007137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:30.551893 14976 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Bootstrap starting.
I20260928 16:47:30.552913 14978 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Bootstrap starting.
I20260928 16:47:30.554492 14976 tablet_bootstrap.cc:654] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:30.555404 14976 log.cc:824] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:30.555454 14978 tablet_bootstrap.cc:654] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:30.556047 14977 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Bootstrap starting.
I20260928 16:47:30.556319 14978 log.cc:824] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:30.557703 14976 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: No bootstrap required, opened a new log
I20260928 16:47:30.557932 14976 ts_tablet_manager.cc:1397] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:47:30.558382 14978 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: No bootstrap required, opened a new log
I20260928 16:47:30.558650 14978 ts_tablet_manager.cc:1397] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:47:30.558769 14977 tablet_bootstrap.cc:654] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:30.559588 14977 log.cc:824] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:30.561674 14977 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: No bootstrap required, opened a new log
I20260928 16:47:30.561887 14977 ts_tablet_manager.cc:1397] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 16:47:30.562120 14976 raft_consensus.cc:348] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.562443 14976 raft_consensus.cc:374] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:30.562525 14976 raft_consensus.cc:729] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27562ccdc1442d68da1ad4688ffb596, State: Initialized, Role: FOLLOWER
I20260928 16:47:30.562484 14978 raft_consensus.cc:348] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.562829 14978 raft_consensus.cc:374] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:30.562892 14978 raft_consensus.cc:729] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb37a9484cc3488dabb1f056f9c06897, State: Initialized, Role: FOLLOWER
I20260928 16:47:30.563014 14976 consensus_queue.cc:260] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.563318 14978 consensus_queue.cc:260] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.564105 14976 ts_tablet_manager.cc:1428] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260928 16:47:30.564256 14978 ts_tablet_manager.cc:1428] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:47:30.564570 14688 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
I20260928 16:47:30.564613 14959 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
I20260928 16:47:30.567378 14977 raft_consensus.cc:348] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.567720 14977 raft_consensus.cc:374] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:30.567793 14977 raft_consensus.cc:729] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c5f1cb96ab4c67b7947f32918a0f2d, State: Initialized, Role: FOLLOWER
I20260928 16:47:30.568236 14977 consensus_queue.cc:260] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.569060 14824 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
I20260928 16:47:30.569363 14977 ts_tablet_manager.cc:1428] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260928 16:47:30.695848 14984 raft_consensus.cc:482] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:30.696125 14984 raft_consensus.cc:504] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.697465 14984 leader_election.cc:288] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
I20260928 16:47:30.701402 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.697727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.30.196.2:55231)
0928 16:47:30.697868 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.697884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.698014 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.698331 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.698334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.698386 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:30.698503 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.698508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.699344 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.699349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.700225 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.700235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.700950 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.700953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.701065 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.701083 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.701172 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.701251 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":19,"spinlock_wait_cycles":49280}
I20260928 16:47:30.701404 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.697814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:55231 (local address 127.30.196.1:45835)
0928 16:47:30.697967 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.697973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.697993 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:47:30.698177 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.698180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.698225 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.698315 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:30.698625 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.699224 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.700355 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.700814 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.700953 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.701012 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.701190 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.701266 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:47:30.701869 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.697999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:39679 (local address 127.30.196.3:37195)
0928 16:47:30.698152 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.698157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.698321 (+ 164us) server_negotiation.cc:407] Connection header received
0928 16:47:30.698513 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.698521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.698574 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.698661 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:30.698951 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.699444 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.700774 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.701187 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.701357 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.701406 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.701655 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.701749 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:47:30.702170 14642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db090b71570e49f39664e2f2c007137d" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27562ccdc1442d68da1ad4688ffb596" is_pre_election: true
I20260928 16:47:30.702270 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.697739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37195 (local address 127.30.196.2:39679)
0928 16:47:30.698217 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.698229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:30.698397 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:30.698673 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:30.698675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:30.698691 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:30.698839 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.698846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.699562 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.699565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.700646 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:30.700657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.701366 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.701370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:30.701476 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.701513 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:47:30.702034 (+ 521us) client_negotiation.cc:240] Negotiation successful
0928 16:47:30.702113 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":48,"spinlock_wait_cycles":4352,"thread_start_us":187,"threads_started":1}
I20260928 16:47:30.702574 14642 raft_consensus.cc:2454] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 0.
I20260928 16:47:30.703049 14912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db090b71570e49f39664e2f2c007137d" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb37a9484cc3488dabb1f056f9c06897" is_pre_election: true
I20260928 16:47:30.703197 14713 leader_election.cc:302] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c5f1cb96ab4c67b7947f32918a0f2d, a27562ccdc1442d68da1ad4688ffb596; no voters:
I20260928 16:47:30.703370 14912 raft_consensus.cc:2454] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 0.
I20260928 16:47:30.703503 14984 raft_consensus.cc:2781] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:30.703584 14984 raft_consensus.cc:482] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:30.703656 14984 raft_consensus.cc:3037] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:30.704879 14984 raft_consensus.cc:504] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.705286 14984 leader_election.cc:288] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 election: Requested vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
I20260928 16:47:30.705717 14642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db090b71570e49f39664e2f2c007137d" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27562ccdc1442d68da1ad4688ffb596"
I20260928 16:47:30.705737 14912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db090b71570e49f39664e2f2c007137d" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb37a9484cc3488dabb1f056f9c06897"
I20260928 16:47:30.705894 14912 raft_consensus.cc:3037] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:30.705834 14642 raft_consensus.cc:3037] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:30.707016 14642 raft_consensus.cc:2454] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 1.
I20260928 16:47:30.707222 14912 raft_consensus.cc:2454] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 1.
I20260928 16:47:30.707381 14713 leader_election.cc:302] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c5f1cb96ab4c67b7947f32918a0f2d, a27562ccdc1442d68da1ad4688ffb596; no voters:
I20260928 16:47:30.707592 14984 raft_consensus.cc:2781] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:30.707901 14984 raft_consensus.cc:686] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 LEADER]: Becoming Leader. State: Replica: 39c5f1cb96ab4c67b7947f32918a0f2d, State: Running, Role: LEADER
I20260928 16:47:30.708283 14984 consensus_queue.cc:237] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:30.711603 14495 catalog_manager.cc:5495] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2). New cstate: current_term: 1 leader_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } health_report { overall_health: HEALTHY } } }
W20260928 16:47:30.726383 14960 tablet.cc:2369] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:30.731452 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.727531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.30.196.2:44131)
0928 16:47:30.727704 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:30.727709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:30.727725 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:47:30.727914 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:30.727916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:30.727977 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:30.728135 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:30.728389 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.728914 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.729800 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:30.730025 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:30.730139 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:30.730564 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:30.730990 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:30.731092 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:30.731225 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:47:30.731281 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:47:30.737358 14912 raft_consensus.cc:1261] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Refusing update from remote peer 39c5f1cb96ab4c67b7947f32918a0f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:30.737555 14642 raft_consensus.cc:1261] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Refusing update from remote peer 39c5f1cb96ab4c67b7947f32918a0f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:30.738091 14984 consensus_queue.cc:1035] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:30.738391 14986 consensus_queue.cc:1035] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:30.745859 14989 mvcc.cc:205] Tried to move back new op lower bound from 7334355151812440064 to 7334355151703584768. Current Snapshot: MvccSnapshot[applied={T|T < 7334355151812440064}]
I20260928 16:47:30.746928 14991 mvcc.cc:205] Tried to move back new op lower bound from 7334355151812440064 to 7334355151703584768. Current Snapshot: MvccSnapshot[applied={T|T < 7334355151812440064}]
I20260928 16:47:30.755617 14493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:30.755823 14493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:30.758304 14493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Sending DeleteTablet for 3 replicas of tablet db090b71570e49f39664e2f2c007137d
I20260928 16:47:30.759140 14621 tablet_service.cc:1514] Processing DeleteTablet for tablet db090b71570e49f39664e2f2c007137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:47954
I20260928 16:47:30.759145 14757 tablet_service.cc:1514] Processing DeleteTablet for tablet db090b71570e49f39664e2f2c007137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:46082
I20260928 16:47:30.759326 14892 tablet_service.cc:1514] Processing DeleteTablet for tablet db090b71570e49f39664e2f2c007137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:45462
I20260928 16:47:30.759822 15002 tablet_replica.cc:330] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: stopping tablet replica
I20260928 16:47:30.760036 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14557
I20260928 16:47:30.762189 15002 raft_consensus.cc:2229] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:30.762560 15002 raft_consensus.cc:2258] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:30.764297 15002 ts_tablet_manager.cc:1905] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:30.767174 15002 ts_tablet_manager.cc:1918] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:30.767277 15002 log.cc:1196] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/db090b71570e49f39664e2f2c007137d
I20260928 16:47:30.767627 15002 ts_tablet_manager.cc:1939] T db090b71570e49f39664e2f2c007137d P eb37a9484cc3488dabb1f056f9c06897: Deleting consensus metadata
I20260928 16:47:30.767616 15001 tablet_replica.cc:330] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: stopping tablet replica
I20260928 16:47:30.768107 15001 raft_consensus.cc:2229] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 LEADER]: Raft consensus shutting down.
W20260928 16:47:30.768460 14482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:45835 (error 108)
I20260928 16:47:30.768728 15001 raft_consensus.cc:2258] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:30.769109 14480 catalog_manager.cc:4841] TS eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): tablet db090b71570e49f39664e2f2c007137d (table test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]) successfully deleted
I20260928 16:47:30.769980 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14693
I20260928 16:47:30.770324 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.769940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.0.0.1:47964)
0928 16:47:30.770142 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.770219 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260928 16:47:30.775869 14482 catalog_manager.cc:4584] TS a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db090b71570e49f39664e2f2c007137d: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45835: connect: Connection refused (error 111)
I20260928 16:47:30.776763 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14829
I20260928 16:47:30.777819 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:30.776921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44131 (local address 127.0.0.1:46098)
0928 16:47:30.777649 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:30.777716 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619}
I20260928 16:47:30.782891 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 14463
I20260928 16:47:50.790031 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--webserver_interface=127.30.196.62
--webserver_port=46611
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:50.902881 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:50.903219 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:50.903322 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:50.907588 15004 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:47:50.907670 15004 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:47:50.907738 15004 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:47:50.907792 15004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:50.907832 15004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:50.907903 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:50.907941 15004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:50.907977 15004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:50.913329 15004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:50.914427 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:50.915529 15004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:50.929280 15011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:50.929389 15004 server_base.cc:1034] running on GCE node
W20260928 16:47:50.929287 15010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:50.929287 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:50.930321 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:50.931574 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:50.932760 15004 hybrid_clock.cc:648] HybridClock initialized: now 1790614070932729 us; error 42 us; skew 500 ppm
Sep 28 16:47:50 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:50.942792 15004 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:47:50.944016 15004 webserver.cc:460] Webserver started at http://127.30.196.62:46611/ using document root <none> and password file <none>
I20260928 16:47:50.944681 15004 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:50.944802 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:50.949150 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:50.951543 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:50.952998 15004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:50.953161 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "9a951d9337fd40a187cee08016e1b46c"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "617a5c7d5f58c2df2f4233f996d532a8"
server_key_iv: "71a39864a19b745858bb82a25c26521d"
server_key_version: "encryptionkey@0"
I20260928 16:47:50.953832 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:50.980444 15004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:50.984030 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:50.984295 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:50.992249 15004 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36217
I20260928 16:47:50.992620 15073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36217 every 8 connection(s)
I20260928 16:47:50.993463 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:50.997701 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15004
I20260928 16:47:50.998262 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:45835
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=35069
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:50.998481 15074 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:51.001116 15074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap starting.
I20260928 16:47:51.021328 15074 log.cc:824] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:51.030270 15074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:51.030689 15074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap complete.
I20260928 16:47:51.035086 15074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:51.035535 15074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.036113 15074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:51.036235 15074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:51.036329 15074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:51.036449 15074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:47:51.038411 15074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:51.038758 15074 leader_election.cc:302] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a951d9337fd40a187cee08016e1b46c; no voters:
I20260928 16:47:51.039099 15074 leader_election.cc:288] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:47:51.039187 15078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:47:51.039364 15078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 LEADER]: Becoming Leader. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Running, Role: LEADER
I20260928 16:47:51.039727 15078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:47:51.040429 15074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:51.040769 15078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a951d9337fd40a187cee08016e1b46c. Latest consensus state: current_term: 2 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:47:51.040890 15078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:51.041205 15079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:47:51.041317 15079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:51.041465 15082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:51.046732 15082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:51.047420 15082 catalog_manager.cc:1259] Loaded cluster ID: 4150fffc4d5348af851196469cba638d
I20260928 16:47:51.047529 15082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:51.050509 15082 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:51.051549 15082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Loaded TSK: 0
I20260928 16:47:51.066615 15082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Generated new TSK 1
I20260928 16:47:51.066944 15082 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:47:51.130059 15076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:51.130358 15076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:51.130446 15076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:51.134670 15076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:51.134792 15076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:51.134913 15076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:47:51.140606 15076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:45835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=35069
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:51.141762 15076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:51.142788 15076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:51.152659 15104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.152649 15101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.152644 15102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:51.153070 15076 server_base.cc:1034] running on GCE node
I20260928 16:47:51.153628 15076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:51.154453 15076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:51.155678 15076 hybrid_clock.cc:648] HybridClock initialized: now 1790614071155631 us; error 44 us; skew 500 ppm
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:51.159201 15076 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:47:51.160256 15076 webserver.cc:460] Webserver started at http://127.30.196.1:35069/ using document root <none> and password file <none>
I20260928 16:47:51.160833 15076 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:51.160909 15076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:51.165056 15076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:47:51.167383 15111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:51.168675 15076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:51.168815 15076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "a27562ccdc1442d68da1ad4688ffb596"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "173fe8044e3c4ad72734219ed84e1076"
server_key_iv: "991e4cc2362078487e87105b25c1d0cd"
server_key_version: "encryptionkey@0"
I20260928 16:47:51.169448 15076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:51.188951 15076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:51.192054 15076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:51.192260 15076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:51.192867 15076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:51.193979 15118 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:47:51.198158 15076 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:47:51.198239 15076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 16:47:51.198335 15076 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:47:51.200094 15076 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:47:51.200156 15076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 16:47:51.200474 15118 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Bootstrap starting.
I20260928 16:47:51.211125 15076 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45835
I20260928 16:47:51.211163 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45835 every 8 connection(s)
I20260928 16:47:51.212164 15076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:51.216311 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15076
I20260928 16:47:51.216814 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:44131
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=37977
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:51.219650 15118 log.cc:824] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:51.227993 15118 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:51.228416 15118 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Bootstrap complete.
I20260928 16:47:51.229832 15118 ts_tablet_manager.cc:1397] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.016s
I20260928 16:47:51.231467 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.214375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:38609)
0928 16:47:51.214903 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.214944 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.215807 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.217133 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.217141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.217502 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:51.217937 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.217951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.219936 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.219940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.221037 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.221047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.222194 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.222199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.222398 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.223152 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:51.223173 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:51.225216 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:51.227898 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.227906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.227928 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.228329 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.228754 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.228759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.228763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.228984 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.229694 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:51.229700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:51.229984 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.230183 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.230505 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":156,"threads_started":1}
I20260928 16:47:51.232017 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.214320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:38609 (local address 127.30.196.62:36217)
0928 16:47:51.214715 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.214735 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.215061 (+ 326us) server_negotiation.cc:407] Connection header received
0928 16:47:51.216221 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.216231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.216563 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.216873 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.218113 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.219719 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.221181 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.222027 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.222219 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.225462 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.225469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.225484 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.225518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.227760 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.228480 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.228487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.228492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.228639 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.229089 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.229095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.229100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.229572 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.229703 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.230320 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.230621 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":165,"threads_started":1}
I20260928 16:47:51.232970 15118 raft_consensus.cc:348] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.233525 15118 raft_consensus.cc:729] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27562ccdc1442d68da1ad4688ffb596, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.234025 15118 consensus_queue.cc:260] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.234781 15227 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
I20260928 16:47:51.235013 15227 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:51.235112 15118 ts_tablet_manager.cc:1428] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260928 16:47:51.235532 15227 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:51.238245 15037 ts_manager.cc:194] Registered new tserver with Master: a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
I20260928 16:47:51.242304 15037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:38609
I20260928 16:47:51.245074 15227 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.1@KRBTEST.COM
I20260928 16:47:51.252764 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.241731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.0.0.1:39068)
0928 16:47:51.242597 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.242643 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.242800 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.243364 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.243370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.243841 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:51.243951 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.243957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.244903 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.244906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.245766 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.245776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.246508 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.246511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.246610 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.247246 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:51.247259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:51.248854 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:51.250966 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.250975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.250980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.251205 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.251523 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.251525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.251527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.251637 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.252231 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:51.252236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:51.252341 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.252467 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.252623 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":204,"threads_started":1}
I20260928 16:47:51.252936 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.241808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.30.196.1:45835)
0928 16:47:51.242267 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.242279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.242738 (+ 459us) server_negotiation.cc:407] Connection header received
0928 16:47:51.242957 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.242965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.243209 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.243364 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.244073 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.244784 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.245890 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.246374 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.246486 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.249007 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.249020 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.249033 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.249061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.250846 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.251318 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.251324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.251325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.251418 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.251800 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.251804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.251805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.252114 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.252254 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.252593 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.252770 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":113,"threads_started":1}
I20260928 16:47:51.254225 15160 tablet_service.cc:1514] Processing DeleteTablet for tablet db090b71570e49f39664e2f2c007137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39068
I20260928 16:47:51.255275 15236 tablet_replica.cc:330] stopping tablet replica
I20260928 16:47:51.255620 15236 raft_consensus.cc:2229] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:51.255954 15236 raft_consensus.cc:2258] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:51.257051 15236 ts_tablet_manager.cc:1905] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:51.260972 15236 ts_tablet_manager.cc:1918] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:51.261063 15236 log.cc:1196] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/db090b71570e49f39664e2f2c007137d
I20260928 16:47:51.261346 15236 ts_tablet_manager.cc:1939] T db090b71570e49f39664e2f2c007137d P a27562ccdc1442d68da1ad4688ffb596: Deleting consensus metadata
I20260928 16:47:51.263463 15024 catalog_manager.cc:4841] TS a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835): tablet db090b71570e49f39664e2f2c007137d (table test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]) successfully deleted
W20260928 16:47:51.344728 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:51.345041 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:51.345122 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:51.349829 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:51.350092 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:51.350206 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:47:51.355901 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:44131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=37977
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:51.357051 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:51.358176 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:51.367856 15244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.367892 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.367965 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:51.368294 15231 server_base.cc:1034] running on GCE node
I20260928 16:47:51.368714 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:51.369397 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:51.370606 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790614071370588 us; error 33 us; skew 500 ppm
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:51.373783 15231 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:47:51.374794 15231 webserver.cc:460] Webserver started at http://127.30.196.2:37977/ using document root <none> and password file <none>
I20260928 16:47:51.375372 15231 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:51.375449 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:51.379608 15231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:51.382038 15251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:51.383373 15231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:47:51.383507 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "39c5f1cb96ab4c67b7947f32918a0f2d"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fc1972c1b7e197273a354f1167749e78"
server_key_iv: "c2a874915fb0b1635fb148b2b57a8712"
server_key_version: "encryptionkey@0"
I20260928 16:47:51.384235 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:51.396787 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:51.400146 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:51.400364 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:51.400990 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:51.402172 15258 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:47:51.406262 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:47:51.406329 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 16:47:51.406445 15231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:47:51.408377 15231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:47:51.408425 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 16:47:51.408700 15258 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Bootstrap starting.
I20260928 16:47:51.414927 15258 log.cc:824] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:51.418218 15231 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:44131
I20260928 16:47:51.418460 15366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:44131 every 8 connection(s)
I20260928 16:47:51.419193 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:51.423281 15258 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:51.423704 15258 tablet_bootstrap.cc:492] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Bootstrap complete.
I20260928 16:47:51.424023 15258 ts_tablet_manager.cc:1397] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260928 16:47:51.424734 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15231
I20260928 16:47:51.425208 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:37195
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=33349
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:51.426959 15258 raft_consensus.cc:348] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.427386 15258 raft_consensus.cc:729] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c5f1cb96ab4c67b7947f32918a0f2d, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.427958 15258 consensus_queue.cc:260] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.429373 15258 ts_tablet_manager.cc:1428] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:51.435228 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.421127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:39567 (local address 127.30.196.62:36217)
0928 16:47:51.421296 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.421301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.421939 (+ 638us) server_negotiation.cc:407] Connection header received
0928 16:47:51.422948 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.422951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.423007 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.423126 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.424449 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.425262 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.426912 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.427375 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.427510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.430599 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.430606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.430613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.430657 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.432707 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.433268 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.433276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.433280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.433406 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.433822 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.433828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.433833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.434136 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.434270 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.434789 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.434966 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:47:51.436029 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.421212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:39567)
0928 16:47:51.421753 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.421807 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.422724 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.423241 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.423248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.423708 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:51.424295 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.424312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.425435 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.425439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.426764 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.426775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.427543 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.427547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.427764 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.428777 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:51.428796 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:51.430413 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:51.432860 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.432865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.432877 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.433151 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.433543 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.433546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.433547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.433696 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.434276 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:51.434282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:51.434527 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.434690 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.435009 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":191,"threads_started":1}
I20260928 16:47:51.437214 15367 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
I20260928 16:47:51.437460 15367 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:51.438020 15367 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:51.439735 15037 ts_manager.cc:194] Registered new tserver with Master: 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
I20260928 16:47:51.441038 15037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:39567
I20260928 16:47:51.443539 15367 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.2@KRBTEST.COM
I20260928 16:47:51.451531 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.440753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44131 (local address 127.0.0.1:41290)
0928 16:47:51.440954 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.440969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.441074 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.442010 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.442013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.442231 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:51.442357 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.442364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.443817 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.443821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.444839 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.444854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.445542 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.445546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.445666 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.446150 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:51.446166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:51.447734 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:51.449816 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.449821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.449824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.450042 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.450365 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.450369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.450370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.450461 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.450956 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:51.450960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:51.451069 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.451182 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.451353 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 16:47:51.451599 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.440880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.30.196.2:44131)
0928 16:47:51.441510 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.441522 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.441544 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:51.441605 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.441610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.441863 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.442008 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.442485 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.443607 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.445079 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.445417 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.445500 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.447880 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.447891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.447913 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.447936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.449686 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.450174 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.450179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.450180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.450258 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.450555 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.450560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.450562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.450815 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.450975 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.451276 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.451441 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":110,"threads_started":1}
I20260928 16:47:51.452616 15300 tablet_service.cc:1514] Processing DeleteTablet for tablet db090b71570e49f39664e2f2c007137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:41290
I20260928 16:47:51.453596 15373 tablet_replica.cc:330] stopping tablet replica
I20260928 16:47:51.453954 15373 raft_consensus.cc:2229] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:51.454252 15373 raft_consensus.cc:2258] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:51.455466 15373 ts_tablet_manager.cc:1905] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:51.460716 15373 ts_tablet_manager.cc:1918] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:51.460835 15373 log.cc:1196] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/db090b71570e49f39664e2f2c007137d
I20260928 16:47:51.461144 15373 ts_tablet_manager.cc:1939] T db090b71570e49f39664e2f2c007137d P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting consensus metadata
I20260928 16:47:51.463251 15021 catalog_manager.cc:4841] TS 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): tablet db090b71570e49f39664e2f2c007137d (table test-table [id=afea2e8a4c1646f2afd9e7e4cd2f9480]) successfully deleted
W20260928 16:47:51.550845 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:51.551126 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:51.551170 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:51.555509 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:51.555583 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:51.555646 15370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:47:51.561407 15370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=33349
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:47:51.562554 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:51.563552 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:51.574087 15378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.574087 15379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:51.574087 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:51.574514 15370 server_base.cc:1034] running on GCE node
I20260928 16:47:51.574960 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:51.575758 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:51.577023 15370 hybrid_clock.cc:648] HybridClock initialized: now 1790614071577001 us; error 32 us; skew 500 ppm
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:51.586877 15370 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:47:51.587986 15370 webserver.cc:460] Webserver started at http://127.30.196.3:33349/ using document root <none> and password file <none>
I20260928 16:47:51.588596 15370 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:51.588690 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:51.592881 15370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:51.595026 15389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:51.596359 15370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:47:51.596455 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "eb37a9484cc3488dabb1f056f9c06897"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fe536b2f4841ef00b286cd1d33e182d8"
server_key_iv: "cb453b4d937e009ddfe4553d4bbf79e4"
server_key_version: "encryptionkey@0"
I20260928 16:47:51.597008 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:51.626434 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:51.629837 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:51.630050 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:51.630662 15370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:51.631493 15370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:51.631569 15370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:51.631642 15370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:51.631727 15370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:51.641739 15370 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:37195
I20260928 16:47:51.641813 15504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:37195 every 8 connection(s)
I20260928 16:47:51.642887 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:51.643476 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15370
I20260928 16:47:51.653159 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.644287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.30.196.62:36217)
0928 16:47:51.644456 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.644460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.644658 (+ 198us) server_negotiation.cc:407] Connection header received
0928 16:47:51.644757 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.644760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.644816 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.644905 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.645893 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.646559 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.647828 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.648114 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.648197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.649815 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.649819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.649821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.649847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.651400 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.651896 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.651900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.651901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.651994 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.652314 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.652316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.652317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.652485 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.652627 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.652883 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.653010 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 16:47:51 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:47:51.658396 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.644914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:58583 (local address 127.30.196.62:36217)
0928 16:47:51.645119 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.645124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.646065 (+ 941us) server_negotiation.cc:407] Connection header received
0928 16:47:51.647642 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.647644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.647739 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.647885 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.649442 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.650197 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.651647 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.652017 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.652111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.654776 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.654780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.654783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.654811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.656408 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.656891 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.656894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.656896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.656976 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.657347 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.657349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.657350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.657512 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.657577 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.658149 (+ 572us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.658254 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260928 16:47:51.659289 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.645418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:58583)
0928 16:47:51.645937 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.645978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.647424 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.648030 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.648037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.648533 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:51.649260 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.649285 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.650352 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.650358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.651514 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.651526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.652174 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.652179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.652416 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.653015 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:51.653039 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:51.654604 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:51.656519 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.656524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.656538 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.656794 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.657073 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:51.657075 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:51.657077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:51.657256 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:51.657635 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:51.657640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:51.657873 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.658095 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.658418 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260928 16:47:51.660627 15505 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36217
I20260928 16:47:51.660951 15505 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:51.661477 15505 heartbeater.cc:511] Master 127.30.196.62:36217 requested a full tablet report, sending...
I20260928 16:47:51.662580 15037 ts_manager.cc:194] Registered new tserver with Master: eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195)
I20260928 16:47:51.663225 15037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:58583
I20260928 16:47:51.666904 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:51.671243 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.667573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.30.196.62:36217)
0928 16:47:51.667791 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.667796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.667916 (+ 120us) server_negotiation.cc:407] Connection header received
0928 16:47:51.668123 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.668125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.668219 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.668332 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:51.668605 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.669140 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.670016 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.670256 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.670368 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.670686 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:51.670819 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 16:47:51.671099 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40172: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 16:47:51.671329 15508 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40172: token expired
W20260928 16:47:51.671707 14962 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.196.62:36217)
I20260928 16:47:51.679277 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.672095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.30.196.62:36217)
0928 16:47:51.672201 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.672205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.672220 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:47:51.672286 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.672288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.672320 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.672379 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:51.672995 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.673515 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.674323 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.674549 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.674660 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.676054 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:51.676057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:51.676060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:51.676085 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:51.677352 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.678044 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.678050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.678052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.678144 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:51.678440 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:51.678443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:51.678445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:51.678618 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:51.678753 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.678990 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.679110 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:47:51.682029 15037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:51.686666 15037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:51.696839 15300 tablet_service.cc:1467] Processing CreateTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 (DEFAULT_TABLE table=test-table [id=d8259854a76542ddb74c53be67e617b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:51.696947 15160 tablet_service.cc:1467] Processing CreateTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 (DEFAULT_TABLE table=test-table [id=d8259854a76542ddb74c53be67e617b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:51.697530 15300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba5a2e36fe14317a76dc8e3fcfcc808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:51.697686 15160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba5a2e36fe14317a76dc8e3fcfcc808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:51.701035 15258 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Bootstrap starting.
I20260928 16:47:51.701104 15118 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Bootstrap starting.
I20260928 16:47:51.702085 15118 tablet_bootstrap.cc:654] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:51.702101 15258 tablet_bootstrap.cc:654] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:51.703816 15118 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: No bootstrap required, opened a new log
I20260928 16:47:51.703908 15118 ts_tablet_manager.cc:1397] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 16:47:51.704411 15118 raft_consensus.cc:348] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.704535 15118 raft_consensus.cc:374] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:51.704571 15118 raft_consensus.cc:729] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27562ccdc1442d68da1ad4688ffb596, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.704689 15118 consensus_queue.cc:260] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.704958 15118 ts_tablet_manager.cc:1428] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:47:51.705871 15258 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: No bootstrap required, opened a new log
I20260928 16:47:51.705971 15258 ts_tablet_manager.cc:1397] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:47:51.706209 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.696084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37195 (local address 127.0.0.1:50976)
0928 16:47:51.696321 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:51.696337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:51.696433 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:51.699851 (+ 3418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:51.699986 (+ 135us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:51.700013 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:51.700197 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.700206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.701773 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.701777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.703586 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:51.703598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.705717 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.705720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:51.705815 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.705884 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 16:47:51.705967 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:47:51.706064 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260928 16:47:51.706455 15258 raft_consensus.cc:348] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.706563 15258 raft_consensus.cc:374] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:51.706607 15258 raft_consensus.cc:729] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c5f1cb96ab4c67b7947f32918a0f2d, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.706754 15258 consensus_queue.cc:260] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.707031 15258 ts_tablet_manager.cc:1428] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:47:51.707237 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:51.696464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.30.196.3:37195)
0928 16:47:51.696829 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:51.696841 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:51.696867 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:47:51.696949 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:51.696955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:51.697177 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:51.697348 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:51.700348 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.701637 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.704228 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:51.705160 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:51.705281 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:51.705458 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:51.706912 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0928 16:47:51.707031 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":153,"threads_started":1}
I20260928 16:47:51.710173 15438 tablet_service.cc:1467] Processing CreateTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 (DEFAULT_TABLE table=test-table [id=d8259854a76542ddb74c53be67e617b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:51.711138 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba5a2e36fe14317a76dc8e3fcfcc808. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 16:47:51.713601 15228 tablet.cc:2369] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:51.719566 15517 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Bootstrap starting.
I20260928 16:47:51.722702 15517 tablet_bootstrap.cc:654] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:51.723712 15517 log.cc:824] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:51.726063 15517 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: No bootstrap required, opened a new log
I20260928 16:47:51.726253 15517 ts_tablet_manager.cc:1397] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:47:51.730083 15517 raft_consensus.cc:348] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.730389 15517 raft_consensus.cc:374] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:51.730451 15517 raft_consensus.cc:729] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb37a9484cc3488dabb1f056f9c06897, State: Initialized, Role: FOLLOWER
I20260928 16:47:51.730931 15517 consensus_queue.cc:260] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:51.731632 15505 heartbeater.cc:503] Master 127.30.196.62:36217 was elected leader, sending a full tablet report...
I20260928 16:47:51.732012 15517 ts_tablet_manager.cc:1428] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 16:47:51.894019 15506 tablet.cc:2369] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:47:51.920469 15368 tablet.cc:2369] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:52.051999 15371 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:52.052313 15371 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:52.053570 15371 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
I20260928 16:47:52.058264 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:52.054272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:48595 (local address 127.30.196.3:37195)
0928 16:47:52.054433 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:52.054439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:52.054461 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:52.054698 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:52.054701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:52.054746 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:52.054838 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:52.055178 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.055785 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057001 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.057588 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057688 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:52.057731 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:52.058061 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 16:47:52.058127 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:47:52.058269 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:52.054218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:55399 (local address 127.30.196.1:45835)
0928 16:47:52.054511 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:52.054515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:52.054746 (+ 231us) server_negotiation.cc:407] Connection header received
0928 16:47:52.054919 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:52.054922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:52.054985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:52.055096 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:52.055358 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.056014 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057151 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.057730 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057848 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:52.057997 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:52.058072 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 16:47:52.058141 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260928 16:47:52.058281 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:52.053998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37195 (local address 127.30.196.2:48595)
0928 16:47:52.054358 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:52.054378 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:52.054519 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:52.054931 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:52.054936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:52.054956 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:52.055057 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:52.055062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.055917 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.055920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:52.056858 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:52.056868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.057707 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:52.057815 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:52.057868 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 16:47:52.058022 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:47:52.058111 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260928 16:47:52.058279 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:52.053998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.30.196.2:55399)
0928 16:47:52.054644 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:52.054655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:52.054787 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:52.055106 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:52.055109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:52.055127 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:52.055238 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:52.055244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.056143 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.056145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:52.057028 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:52.057038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.057866 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.057869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:52.057935 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:52.057952 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:47:52.058050 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:47:52.058127 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":395,"thread_start_us":57,"threads_started":1}
I20260928 16:47:52.059027 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba5a2e36fe14317a76dc8e3fcfcc808" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27562ccdc1442d68da1ad4688ffb596" is_pre_election: true
I20260928 16:47:52.059074 15458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba5a2e36fe14317a76dc8e3fcfcc808" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb37a9484cc3488dabb1f056f9c06897" is_pre_election: true
I20260928 16:47:52.059367 15180 raft_consensus.cc:2454] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 0.
I20260928 16:47:52.059366 15458 raft_consensus.cc:2454] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 0.
I20260928 16:47:52.059895 15253 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897; no voters:
I20260928 16:47:52.060177 15371 raft_consensus.cc:2781] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:52.060262 15371 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:52.060305 15371 raft_consensus.cc:3037] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:52.061398 15371 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:52.061815 15371 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 election: Requested vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835)
I20260928 16:47:52.062134 15458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba5a2e36fe14317a76dc8e3fcfcc808" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb37a9484cc3488dabb1f056f9c06897"
I20260928 16:47:52.062136 15180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba5a2e36fe14317a76dc8e3fcfcc808" candidate_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27562ccdc1442d68da1ad4688ffb596"
I20260928 16:47:52.062285 15458 raft_consensus.cc:3037] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:52.062299 15180 raft_consensus.cc:3037] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:52.063290 15458 raft_consensus.cc:2454] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 1.
I20260928 16:47:52.063297 15180 raft_consensus.cc:2454] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c5f1cb96ab4c67b7947f32918a0f2d in term 1.
I20260928 16:47:52.063591 15253 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897; no voters:
I20260928 16:47:52.063791 15371 raft_consensus.cc:2781] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:52.063982 15371 raft_consensus.cc:686] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 LEADER]: Becoming Leader. State: Replica: 39c5f1cb96ab4c67b7947f32918a0f2d, State: Running, Role: LEADER
I20260928 16:47:52.064248 15371 consensus_queue.cc:237] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:47:52.067086 15036 catalog_manager.cc:5495] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2). New cstate: current_term: 1 leader_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } health_report { overall_health: HEALTHY } } }
I20260928 16:47:52.158962 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:52.154917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.30.196.2:44131)
0928 16:47:52.155201 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:52.155207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:52.155219 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:47:52.155293 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:52.155295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:52.155348 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:52.155485 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:52.155773 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.156362 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.157220 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:52.157459 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:52.157586 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:52.157960 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:52.158428 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:52.158516 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:52.158651 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 16:47:52.158720 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
I20260928 16:47:52.163137 15180 raft_consensus.cc:1261] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Refusing update from remote peer 39c5f1cb96ab4c67b7947f32918a0f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:52.163924 15458 raft_consensus.cc:1261] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Refusing update from remote peer 39c5f1cb96ab4c67b7947f32918a0f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:52.163964 15371 consensus_queue.cc:1035] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:52.164525 15371 consensus_queue.cc:1035] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:52.169584 15526 mvcc.cc:205] Tried to move back new op lower bound from 7334355239573368832 to 7334355239177699328. Current Snapshot: MvccSnapshot[applied={T|T < 7334355239573368832}]
I20260928 16:47:52.171972 15528 mvcc.cc:205] Tried to move back new op lower bound from 7334355239573368832 to 7334355239177699328. Current Snapshot: MvccSnapshot[applied={T|T < 7334355239573368832}]
I20260928 16:47:52.178460 15037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:52.178615 15037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:52.180970 15037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Sending DeleteTablet for 3 replicas of tablet aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:47:52.181627 15160 tablet_service.cc:1514] Processing DeleteTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:52 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39068
I20260928 16:47:52.181707 15438 tablet_service.cc:1514] Processing DeleteTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:52 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:50976
I20260928 16:47:52.181954 15300 tablet_service.cc:1514] Processing DeleteTablet for tablet aba5a2e36fe14317a76dc8e3fcfcc808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:52 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:41290
I20260928 16:47:52.182017 15535 tablet_replica.cc:330] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: stopping tablet replica
I20260928 16:47:52.182154 15535 raft_consensus.cc:2229] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:52.182293 15535 raft_consensus.cc:2258] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:52.182471 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15076
I20260928 16:47:52.183326 15537 tablet_replica.cc:330] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: stopping tablet replica
I20260928 16:47:52.183490 15537 raft_consensus.cc:2229] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:52.183853 15537 raft_consensus.cc:2258] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:52.184612 15537 ts_tablet_manager.cc:1905] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:52.184868 15536 tablet_replica.cc:330] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: stopping tablet replica
I20260928 16:47:52.185232 15536 raft_consensus.cc:2229] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:52.185640 15536 raft_consensus.cc:2258] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:52.187317 15536 ts_tablet_manager.cc:1905] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:52.188218 15537 ts_tablet_manager.cc:1918] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:52.188306 15537 log.cc:1196] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:47:52.188606 15537 ts_tablet_manager.cc:1939] T aba5a2e36fe14317a76dc8e3fcfcc808 P 39c5f1cb96ab4c67b7947f32918a0f2d: Deleting consensus metadata
I20260928 16:47:52.189590 15021 catalog_manager.cc:4841] TS 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): tablet aba5a2e36fe14317a76dc8e3fcfcc808 (table test-table [id=d8259854a76542ddb74c53be67e617b9]) successfully deleted
I20260928 16:47:52.190553 15536 ts_tablet_manager.cc:1918] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:52.190651 15536 log.cc:1196] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:47:52.190966 15536 ts_tablet_manager.cc:1939] T aba5a2e36fe14317a76dc8e3fcfcc808 P eb37a9484cc3488dabb1f056f9c06897: Deleting consensus metadata
W20260928 16:47:52.191222 15024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:45835 (error 108)
I20260928 16:47:52.192256 15022 catalog_manager.cc:4841] TS eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): tablet aba5a2e36fe14317a76dc8e3fcfcc808 (table test-table [id=d8259854a76542ddb74c53be67e617b9]) successfully deleted
W20260928 16:47:52.192571 15234 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:47:52.192238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:45835 (local address 127.0.0.1:39070)
0928 16:47:52.192374 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:52.192458 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45835: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:47:52.192791 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15231
W20260928 16:47:52.192849 15024 catalog_manager.cc:4584] TS a27562ccdc1442d68da1ad4688ffb596 (127.30.196.1:45835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aba5a2e36fe14317a76dc8e3fcfcc808: Network error: Client connection negotiation failed: client connection to 127.30.196.1:45835: connect: Connection reset by peer (error 104)
I20260928 16:47:52.199187 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15370
I20260928 16:47:52.205152 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15004
I20260928 16:48:12.211823 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--webserver_interface=127.30.196.62
--webserver_port=46611
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:12.330621 15539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:12.330928 15539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:12.330979 15539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:12.335285 15539 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:48:12.335357 15539 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:48:12.335373 15539 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:48:12.335387 15539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:12.335405 15539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:12.335458 15539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:12.335492 15539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:12.335527 15539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:12.341750 15539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:12.342922 15539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:12.344059 15539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:48:12.353452 15539 server_base.cc:1034] running on GCE node
W20260928 16:48:12.353488 15544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:12.353675 15547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:12.353488 15545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:12.354557 15539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:12.355866 15539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:12.357065 15539 hybrid_clock.cc:648] HybridClock initialized: now 1790614092357038 us; error 44 us; skew 500 ppm
Sep 28 16:48:12 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614092, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:12.360384 15539 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:12.361498 15539 webserver.cc:460] Webserver started at http://127.30.196.62:46611/ using document root <none> and password file <none>
I20260928 16:48:12.362106 15539 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:12.362206 15539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:12.366662 15539 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:12.369446 15554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:12.370965 15539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:12.371150 15539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "9a951d9337fd40a187cee08016e1b46c"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "617a5c7d5f58c2df2f4233f996d532a8"
server_key_iv: "71a39864a19b745858bb82a25c26521d"
server_key_version: "encryptionkey@0"
I20260928 16:48:12.371857 15539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:12.405421 15539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:12.408828 15539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:12.409087 15539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:12.416746 15539 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36217
I20260928 16:48:12.417097 15607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36217 every 8 connection(s)
I20260928 16:48:12.417922 15539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:12.419462 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15539
I20260928 16:48:12.419976 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:45835
--local_ip_for_outbound_sockets=127.30.196.1
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=35069
--webserver_interface=127.30.196.1
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:48:12.422760 15608 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:48:12.424606 15608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap starting.
I20260928 16:48:12.442732 15608 log.cc:824] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:12.454681 15608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:48:12.455149 15608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Bootstrap complete.
I20260928 16:48:12.459306 15608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:48:12.459805 15608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Initialized, Role: FOLLOWER
I20260928 16:48:12.460359 15608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:48:12.460511 15608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:12.460605 15608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:12.460705 15608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:48:12.462646 15608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:48:12.462996 15608 leader_election.cc:302] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a951d9337fd40a187cee08016e1b46c; no voters:
I20260928 16:48:12.463318 15608 leader_election.cc:288] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:48:12.463495 15612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:48:12.463846 15612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [term 3 LEADER]: Becoming Leader. State: Replica: 9a951d9337fd40a187cee08016e1b46c, State: Running, Role: LEADER
I20260928 16:48:12.464351 15612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } }
I20260928 16:48:12.464708 15608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:12.465991 15614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a951d9337fd40a187cee08016e1b46c. Latest consensus state: current_term: 3 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:48:12.466044 15613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a951d9337fd40a187cee08016e1b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a951d9337fd40a187cee08016e1b46c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36217 } } }
I20260928 16:48:12.466157 15613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:12.466110 15614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:12.466647 15622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:12.472012 15622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:12.472669 15622 catalog_manager.cc:1259] Loaded cluster ID: 4150fffc4d5348af851196469cba638d
I20260928 16:48:12.472730 15622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:12.475796 15622 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:12.476619 15622 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Loaded TSK: 0
I20260928 16:48:12.476708 15622 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a951d9337fd40a187cee08016e1b46c: Loaded TSK: 1
F20260928 16:48:12.477481 15622 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f024c7ba9ad google::LogMessage::Fail() at ??:0
@ 0x7f024c7bd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f024c7ba4de google::LogMessage::Flush() at ??:0
@ 0x7f024c7bba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f024f47e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f024f476f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f02571237bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f02570fd99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f02570fe178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f025713fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f025715e82b std::function<>::operator()() at ??:0
@ 0x7f02570fdb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f02570fef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f02570fb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f025713f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f024ea0c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f024ea0cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f024ea0e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f024e9fd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02500416db start_thread at ??:0
@ 0x7f024b2b861f clone at ??:0
W20260928 16:48:12.584219 15610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:12.584699 15610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:12.584774 15610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:12.591355 15610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:12.591423 15610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:12.591522 15610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:12.597519 15610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:45835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=35069
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:12.598703 15610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:12.599877 15610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:12.610164 15656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:12.610152 15655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:12.610220 15658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:12.610430 15610 server_base.cc:1034] running on GCE node
I20260928 16:48:12.610859 15610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:12.611588 15610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:12.612776 15610 hybrid_clock.cc:648] HybridClock initialized: now 1790614092612745 us; error 50 us; skew 500 ppm
Sep 28 16:48:12 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614092, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:12.615743 15610 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:48:12.616729 15610 webserver.cc:460] Webserver started at http://127.30.196.1:35069/ using document root <none> and password file <none>
I20260928 16:48:12.617312 15610 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:12.617409 15610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:12.621620 15610 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:12.624086 15665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:12.625737 15610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:48:12.625869 15610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "a27562ccdc1442d68da1ad4688ffb596"
format_stamp: "Formatted at 2026-09-28 16:47:29 on dist-test-slave-5rg1"
server_key: "173fe8044e3c4ad72734219ed84e1076"
server_key_iv: "991e4cc2362078487e87105b25c1d0cd"
server_key_version: "encryptionkey@0"
I20260928 16:48:12.626557 15610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:12.656751 15610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:12.660439 15610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:12.660684 15610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:12.661317 15610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:12.662503 15672 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:48:12.668175 15610 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:48:12.668292 15610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 16:48:12.668387 15610 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:48:12.670200 15610 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:48:12.670286 15610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:48:12.670611 15672 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Bootstrap starting.
I20260928 16:48:12.681383 15610 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:45835
I20260928 16:48:12.681598 15780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:45835 every 8 connection(s)
I20260928 16:48:12.682469 15610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:12.688014 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15610
I20260928 16:48:12.688516 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:44131
--local_ip_for_outbound_sockets=127.30.196.2
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=37977
--webserver_interface=127.30.196.2
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:48:12.695108 15672 log.cc:824] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:12.702628 15672 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:48:12.703060 15672 tablet_bootstrap.cc:492] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Bootstrap complete.
I20260928 16:48:12.704313 15672 ts_tablet_manager.cc:1397] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.004s
I20260928 16:48:12.706832 15672 raft_consensus.cc:348] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:12.707293 15672 raft_consensus.cc:729] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27562ccdc1442d68da1ad4688ffb596, State: Initialized, Role: FOLLOWER
I20260928 16:48:12.707865 15672 consensus_queue.cc:260] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:12.708956 15672 ts_tablet_manager.cc:1428] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260928 16:48:12.719579 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:48:12.684714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:47447)
0928 16:48:12.685239 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.685294 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:12.686144 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:12.718780 (+ 32636us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":158,"threads_started":1}
W20260928 16:48:12.720249 15781 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:36217 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:36217: Client connection negotiation failed: client connection to 127.30.196.62:36217: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:48:12.721005 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.720730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:59139)
0928 16:48:12.720862 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.720936 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:48:12.721807 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.721566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:55973)
0928 16:48:12.721689 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.721746 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:48:12.722009 15781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:48:12.814759 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:12.815052 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:12.815099 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:12.819609 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:12.819722 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:12.819796 15784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:12.825213 15784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:44131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=37977
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:12.826385 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:12.827426 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:12.838307 15794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:12.838353 15784 server_base.cc:1034] running on GCE node
W20260928 16:48:12.838307 15791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:12.838307 15792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:12.838985 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:12.839810 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:12.840984 15784 hybrid_clock.cc:648] HybridClock initialized: now 1790614092840963 us; error 36 us; skew 500 ppm
Sep 28 16:48:12 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614092, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:12.843888 15784 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:48:12.844822 15784 webserver.cc:460] Webserver started at http://127.30.196.2:37977/ using document root <none> and password file <none>
I20260928 16:48:12.845423 15784 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:12.845490 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:12.849610 15784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:12.851820 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:12.853140 15784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:12.853284 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "39c5f1cb96ab4c67b7947f32918a0f2d"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fc1972c1b7e197273a354f1167749e78"
server_key_iv: "c2a874915fb0b1635fb148b2b57a8712"
server_key_version: "encryptionkey@0"
I20260928 16:48:12.853848 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:12.868377 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:12.872005 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:12.872236 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:12.872867 15784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:12.873718 15784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:12.873793 15784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:12.873868 15784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:12.873916 15784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:12.883275 15784 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:44131
I20260928 16:48:12.883299 15916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:44131 every 8 connection(s)
I20260928 16:48:12.884352 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:12.887179 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15784
I20260928 16:48:12.887733 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:37195
--local_ip_for_outbound_sockets=127.30.196.3
--tserver_master_addrs=127.30.196.62:36217
--webserver_port=33349
--webserver_interface=127.30.196.3
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:48:12.889084 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.886540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:54317)
0928 16:48:12.887172 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.887330 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
W20260928 16:48:12.889672 15917 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:36217 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:36217: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
I20260928 16:48:12.890458 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.890132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:53249)
0928 16:48:12.890284 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.890362 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:48:12.891258 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.890952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:54777)
0928 16:48:12.891114 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.891175 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:48:12.891455 15917 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:48:13.007305 15920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:13.007727 15920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:13.007805 15920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:13.012441 15920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:13.012511 15920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:13.012583 15920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:13.018750 15920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=33349
--tserver_master_addrs=127.30.196.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:13.019934 15920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:13.020995 15920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:13.031723 15928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:13.031745 15920 server_base.cc:1034] running on GCE node
W20260928 16:48:13.031706 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:13.031653 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:13.032392 15920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:13.033077 15920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:13.034264 15920 hybrid_clock.cc:648] HybridClock initialized: now 1790614093034252 us; error 25 us; skew 500 ppm
Sep 28 16:48:13 dist-test-slave-5rg1 krb5kdc[14450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614093, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:13.037317 15920 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:48:13.038220 15920 webserver.cc:460] Webserver started at http://127.30.196.3:33349/ using document root <none> and password file <none>
I20260928 16:48:13.038806 15920 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:13.038875 15920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:13.042932 15920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:13.045323 15935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:13.046708 15920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:13.046865 15920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "eb37a9484cc3488dabb1f056f9c06897"
format_stamp: "Formatted at 2026-09-28 16:47:30 on dist-test-slave-5rg1"
server_key: "fe536b2f4841ef00b286cd1d33e182d8"
server_key_iv: "cb453b4d937e009ddfe4553d4bbf79e4"
server_key_version: "encryptionkey@0"
I20260928 16:48:13.047451 15920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:13.069444 15920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:13.072867 15920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:13.073107 15920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:13.073753 15920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:13.074640 15920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:13.074715 15920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:13.074790 15920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:13.074836 15920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:13.084201 15920 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:37195
I20260928 16:48:13.084239 16049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:37195 every 8 connection(s)
I20260928 16:48:13.085282 15920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:13.086349 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 15920
I20260928 16:48:13.089497 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.087648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:40917)
0928 16:48:13.088174 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.088281 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
W20260928 16:48:13.089972 16050 heartbeater.cc:644] Failed to heartbeat to 127.30.196.62:36217 (0 consecutive failures): Network error: Failed to ping master at 127.30.196.62:36217: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
I20260928 16:48:13.090616 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.090369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:34959)
0928 16:48:13.090511 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.090560 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:48:13.091246 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.091065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:47679)
0928 16:48:13.091157 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.091198 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 16:48:13.091429 16050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:48:13.723232 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.722739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:58493)
0928 16:48:13.723044 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.723113 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260928 16:48:13.892541 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.892124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:50933)
0928 16:48:13.892367 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.892422 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260928 16:48:14.092650 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.092187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:35577)
0928 16:48:14.092487 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.092543 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260928 16:48:14.247141 16057 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:14.247427 16057 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:14.248972 16057 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
Sep 28 16:48:14 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614092, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.3@KRBTEST.COM
Sep 28 16:48:14 dist-test-slave-5rg1 krb5kdc[14450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614092, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.2@KRBTEST.COM
I20260928 16:48:14.268695 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.249267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:44131 (local address 127.30.196.1:52417)
0928 16:48:14.249775 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.249786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:14.249870 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:14.253239 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:14.253243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:14.253449 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:14.253569 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:14.253578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.255787 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.255792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:14.257737 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:14.257755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.259806 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.259811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:14.259967 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:14.260489 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:14.260501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:14.262111 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:14.265618 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:14.265620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:14.265621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:14.265800 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:14.266271 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:14.266274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:14.266276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:14.266359 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:14.267238 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:14.267247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:14.267691 (+ 444us) client_negotiation.cc:769] Sending connection context
0928 16:48:14.268020 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 16:48:14.268445 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":256,"spinlock_wait_cycles":2176,"thread_start_us":67,"threads_started":1}
I20260928 16:48:14.268838 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.249356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:35531 (local address 127.30.196.3:37195)
0928 16:48:14.249969 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:14.249991 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:14.250026 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:48:14.250942 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:14.250953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:14.251485 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:14.251906 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:14.253768 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.255486 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.258005 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.258839 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.259057 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:14.262292 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:14.262299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:14.262314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:14.262351 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:14.264981 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:14.265649 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:14.265656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:14.265660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:14.265804 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:14.266221 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:14.266228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:14.266233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:14.267063 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:14.267244 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:14.268287 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0928 16:48:14.268613 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":144,"threads_started":1}
I20260928 16:48:14.268850 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.249419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:52417 (local address 127.30.196.2:44131)
0928 16:48:14.249693 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:14.249720 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:14.249893 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:48:14.250960 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:14.250969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:14.251485 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:14.251906 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:14.253768 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.255486 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.258005 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.258839 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.259057 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:14.262339 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:14.262344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:14.262357 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:14.262391 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:14.264981 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:14.266002 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:14.266007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:14.266010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:14.266134 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:14.266510 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:14.266514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:14.266516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:14.267063 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:14.267244 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:14.268287 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0928 16:48:14.268613 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260928 16:48:14.268695 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.249092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:37195 (local address 127.30.196.1:35531)
0928 16:48:14.249430 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.249446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:14.249557 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:14.252083 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:14.252090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:14.252705 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:14.253469 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:14.253493 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.255787 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.255792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:14.257737 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:14.257755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:14.259077 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:14.259081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:14.259332 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:14.260149 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:14.260168 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:14.261930 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:14.265161 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:14.265169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:14.265184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:14.265506 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:14.265940 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:14.265943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:14.265944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:14.266106 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:14.267238 (+ 1132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:14.267247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:14.267689 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 16:48:14.267966 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 16:48:14.268512 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":462,"spinlock_wait_cycles":18560,"thread_start_us":79,"threads_started":1}
W20260928 16:48:14.272008 15666 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
W20260928 16:48:14.272739 15667 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:48:14.272845 15667 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a27562ccdc1442d68da1ad4688ffb596; no voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897
I20260928 16:48:14.273155 16057 raft_consensus.cc:2726] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:48:14.724632 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.724241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:53063)
0928 16:48:14.724433 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.724516 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:48:14.893919 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.893453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:36541)
0928 16:48:14.893741 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.893806 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260928 16:48:15.093940 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.093481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:54001)
0928 16:48:15.093780 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:15.093841 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
I20260928 16:48:15.726020 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.725549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:43093)
0928 16:48:15.725835 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:15.725904 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260928 16:48:15.895402 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.894891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:46141)
0928 16:48:15.895215 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:15.895297 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260928 16:48:16.061671 16066 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:16.061810 16066 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:16.062264 16066 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
W20260928 16:48:16.063143 15667 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
W20260928 16:48:16.063236 15666 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:48:16.063357 15666 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a27562ccdc1442d68da1ad4688ffb596; no voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897
I20260928 16:48:16.063525 16066 raft_consensus.cc:2726] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:48:16.095315 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:16.094819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:38439)
0928 16:48:16.095149 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:16.095216 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260928 16:48:16.727555 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:16.727039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:56983)
0928 16:48:16.727377 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:16.727454 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
I20260928 16:48:16.896929 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:16.896416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:38327)
0928 16:48:16.896726 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:16.896790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260928 16:48:17.096520 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:17.096117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:60695)
0928 16:48:17.096352 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:17.096427 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260928 16:48:17.729225 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:17.728708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:48211)
0928 16:48:17.729059 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:17.729124 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":89,"threads_started":1}
I20260928 16:48:17.898368 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:17.897837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:51413)
0928 16:48:17.898160 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:17.898229 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260928 16:48:18.097703 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.097254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:36451)
0928 16:48:18.097540 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:18.097593 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260928 16:48:18.701535 16074 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:18.701678 16074 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:18.702224 16074 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
W20260928 16:48:18.703117 15667 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
W20260928 16:48:18.703217 15666 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:48:18.703275 15666 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a27562ccdc1442d68da1ad4688ffb596; no voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897
I20260928 16:48:18.703424 16074 raft_consensus.cc:2726] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:48:18.730530 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.730068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:54029)
0928 16:48:18.730356 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:18.730422 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260928 16:48:18.900004 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.899474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:57065)
0928 16:48:18.899833 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:18.899902 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260928 16:48:19.098958 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.098480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:32835)
0928 16:48:19.098763 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.098837 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260928 16:48:19.732038 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.731499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:48303)
0928 16:48:19.731822 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.731890 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260928 16:48:19.901247 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.900795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:41091)
0928 16:48:19.901092 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.901147 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":75,"threads_started":1}
I20260928 16:48:20.100374 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:20.099816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:50397)
0928 16:48:20.100209 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:20.100276 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":89,"threads_started":1}
I20260928 16:48:20.733161 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:20.732756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:36989)
0928 16:48:20.733001 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:20.733060 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260928 16:48:20.902475 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:20.902053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:42087)
0928 16:48:20.902317 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:20.902376 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260928 16:48:21.101668 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.101175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:50689)
0928 16:48:21.101499 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.101568 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":126,"threads_started":1}
I20260928 16:48:21.734365 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.733947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:60271)
0928 16:48:21.734206 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.734265 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260928 16:48:21.904183 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.903492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:38317)
0928 16:48:21.903933 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.904027 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":103,"threads_started":1}
I20260928 16:48:22.103006 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.102494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:36229)
0928 16:48:22.102772 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.102839 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260928 16:48:22.735940 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.735393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:39259)
0928 16:48:22.735764 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.735831 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260928 16:48:22.905624 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.905132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:45293)
0928 16:48:22.905457 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.905518 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":138,"threads_started":1}
I20260928 16:48:22.973124 16089 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:22.973304 16089 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:22.973799 16089 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
W20260928 16:48:22.974665 15667 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
W20260928 16:48:22.974767 15666 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:48:22.974957 15666 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a27562ccdc1442d68da1ad4688ffb596; no voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897
I20260928 16:48:22.975129 16089 raft_consensus.cc:2726] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:48:23.104524 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:23.104018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:48549)
0928 16:48:23.104349 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:23.104421 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260928 16:48:23.737319 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:23.736790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:49303)
0928 16:48:23.737138 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:23.737215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260928 16:48:23.907014 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:23.906556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:59357)
0928 16:48:23.906854 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:23.906905 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260928 16:48:24.106130 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:24.105517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:35245)
0928 16:48:24.105840 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:24.105953 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":80,"threads_started":1}
I20260928 16:48:24.738721 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:24.738236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:34809)
0928 16:48:24.738546 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:24.738621 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260928 16:48:24.751261 16095 raft_consensus.cc:482] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:24.751433 16095 raft_consensus.cc:504] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb37a9484cc3488dabb1f056f9c06897" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 37195 } } peers { permanent_uuid: "a27562ccdc1442d68da1ad4688ffb596" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 45835 } } peers { permanent_uuid: "39c5f1cb96ab4c67b7947f32918a0f2d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 44131 } }
I20260928 16:48:24.752022 16095 leader_election.cc:288] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195), 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131)
W20260928 16:48:24.752873 15667 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb37a9484cc3488dabb1f056f9c06897 (127.30.196.3:37195): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
W20260928 16:48:24.752988 15666 leader_election.cc:341] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39c5f1cb96ab4c67b7947f32918a0f2d (127.30.196.2:44131): Not found: Tablet not found: aba5a2e36fe14317a76dc8e3fcfcc808
I20260928 16:48:24.753108 15666 leader_election.cc:302] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a27562ccdc1442d68da1ad4688ffb596; no voters: 39c5f1cb96ab4c67b7947f32918a0f2d, eb37a9484cc3488dabb1f056f9c06897
I20260928 16:48:24.753265 16095 raft_consensus.cc:2726] T aba5a2e36fe14317a76dc8e3fcfcc808 P a27562ccdc1442d68da1ad4688ffb596 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:48:24.908458 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:24.907969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:57925)
0928 16:48:24.908261 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:24.908328 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260928 16:48:25.107643 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.107103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:58507)
0928 16:48:25.107465 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:25.107540 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
I20260928 16:48:25.740459 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.739851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:58997)
0928 16:48:25.740231 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:25.740302 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20260928 16:48:25.909822 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.909366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:35639)
0928 16:48:25.909665 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:25.909723 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260928 16:48:26.109061 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.108586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:33919)
0928 16:48:26.108891 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.108955 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260928 16:48:26.741946 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.741500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:41319)
0928 16:48:26.741791 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.741846 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":130,"threads_started":1}
I20260928 16:48:26.911144 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.910665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:43481)
0928 16:48:26.910977 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.911039 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260928 16:48:27.110498 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:27.109894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.3:33447)
0928 16:48:27.110304 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:27.110369 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":142,"threads_started":1}
I20260928 16:48:27.743621 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:27.743111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.1:38577)
0928 16:48:27.743446 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:27.743518 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":89,"threads_started":1}
I20260928 16:48:27.912814 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:27.912308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36217 (local address 127.30.196.2:55991)
0928 16:48:27.912607 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:27.912663 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.62:36217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
I20260928 16:48:28.089406 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15610
I20260928 16:48:28.096652 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15784
I20260928 16:48:28.102780 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 15920
2026-09-28T16:48:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59741 ms)
[----------] 30 tests from SecurityITest (248487 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:28 dist-test-slave-5rg1 krb5kdc[16108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:28 dist-test-slave-5rg1 krb5kdc[16108](info): set up 2 sockets
Sep 28 16:48:28 dist-test-slave-5rg1 krb5kdc[16108](info): commencing operation
krb5kdc: starting...
W20260928 16:48:29.180164 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:48:29.311041 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:39747
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:29.424259 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:29.424608 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:29.424696 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:29.429381 16122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:29.429476 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:29.429517 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:29.429569 16122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:29.429621 16122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:29.435165 16122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:29.436440 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:29.437507 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:48:29.446866 16122 server_base.cc:1034] running on GCE node
W20260928 16:48:29.446969 16128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.447060 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.446969 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.447753 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:29.449046 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:29.450253 16122 hybrid_clock.cc:648] HybridClock initialized: now 1790614109450225 us; error 37 us; skew 500 ppm
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:29.453311 16122 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:29.454335 16122 webserver.cc:460] Webserver started at http://127.30.196.62:36675/ using document root <none> and password file <none>
I20260928 16:48:29.454891 16122 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:29.454952 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:29.455124 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:29.457263 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "49f74f12ab9d4854bf40cef5d710846a"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "4177d1d67f6a0a4443bd82cb1601a3a6"
server_key_iv: "9d7a2c6349837f9f66afec93b6a0385b"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.457790 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "49f74f12ab9d4854bf40cef5d710846a"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "4177d1d67f6a0a4443bd82cb1601a3a6"
server_key_iv: "9d7a2c6349837f9f66afec93b6a0385b"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.461513 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:48:29.463888 16137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.465087 16122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:29.465214 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "49f74f12ab9d4854bf40cef5d710846a"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "4177d1d67f6a0a4443bd82cb1601a3a6"
server_key_iv: "9d7a2c6349837f9f66afec93b6a0385b"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.465317 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:29.481069 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:29.484279 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:29.484481 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:29.491551 16122 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:39747
I20260928 16:48:29.491576 16190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:39747 every 8 connection(s)
I20260928 16:48:29.492527 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:29.495366 16191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:29.497983 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16122
I20260928 16:48:29.498085 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:29.498378 31504 external_mini_cluster.cc:1428] Setting key 6b5dfbfc5540206e6997a8e13c2b898c
I20260928 16:48:29.501703 16191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: Bootstrap starting.
I20260928 16:48:29.503998 16191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:29.504750 16191 log.cc:824] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:29.506711 16191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: No bootstrap required, opened a new log
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:29.509929 16191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } }
I20260928 16:48:29.510149 16191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:29.510207 16191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f74f12ab9d4854bf40cef5d710846a, State: Initialized, Role: FOLLOWER
I20260928 16:48:29.510720 16191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } }
I20260928 16:48:29.510876 16191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:29.510957 16191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:29.511070 16191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:29.512109 16191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } }
I20260928 16:48:29.512476 16191 leader_election.cc:302] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f74f12ab9d4854bf40cef5d710846a; no voters:
I20260928 16:48:29.512797 16191 leader_election.cc:288] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:29.512912 16196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:29.513095 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.499873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.30.196.62:39747)
0928 16:48:29.500327 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.500346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.500384 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:48:29.501004 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.501021 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.501349 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.501682 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.502490 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.503472 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.504447 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.504954 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.505128 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.507834 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.507839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.507852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:29.507937 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.510180 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.510631 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.510636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.510640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.510751 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.511024 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.511029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.511030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.511421 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.511634 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.511924 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.512203 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260928 16:48:29.513113 16196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [term 1 LEADER]: Becoming Leader. State: Replica: 49f74f12ab9d4854bf40cef5d710846a, State: Running, Role: LEADER
I20260928 16:48:29.513474 16196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } }
I20260928 16:48:29.514030 16191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:29.514652 16198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f74f12ab9d4854bf40cef5d710846a. Latest consensus state: current_term: 1 leader_uuid: "49f74f12ab9d4854bf40cef5d710846a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } } }
I20260928 16:48:29.514881 16198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:29.515074 16197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f74f12ab9d4854bf40cef5d710846a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f74f12ab9d4854bf40cef5d710846a" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 39747 } } }
I20260928 16:48:29.515221 16197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:29.515769 16204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:29.519016 16204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:29.524466 16204 catalog_manager.cc:1347] Generated new cluster ID: d765e0aab40c4f53bf0e90cb0d819798
I20260928 16:48:29.524531 16204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:29.530903 16204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:29.532147 16204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:29.537447 16204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: Generated new TSK 0
I20260928 16:48:29.538249 16204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:48:29.575439 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:29.691393 16218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:29.691731 16218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:29.691824 16218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:29.695940 16218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:29.696039 16218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:29.696141 16218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:29.701486 16218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:29.702590 16218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:29.703684 16218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:29.714212 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.714283 16218 server_base.cc:1034] running on GCE node
W20260928 16:48:29.714223 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.714215 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.714980 16218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:29.715734 16218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:29.716946 16218 hybrid_clock.cc:648] HybridClock initialized: now 1790614109716934 us; error 72 us; skew 500 ppm
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:29.719888 16218 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:48:29.720803 16218 webserver.cc:460] Webserver started at http://127.30.196.1:40339/ using document root <none> and password file <none>
I20260928 16:48:29.721431 16218 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:29.721524 16218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:29.721750 16218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:29.723897 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "db6eb9c4007d461e81b9a51df542e26b"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "7ec9a80745285083156a84877ce75a45"
server_key_iv: "8a300effb18ae285284cfd7e142848f4"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.724468 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "db6eb9c4007d461e81b9a51df542e26b"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "7ec9a80745285083156a84877ce75a45"
server_key_iv: "8a300effb18ae285284cfd7e142848f4"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.728477 16218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:48:29.731118 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.732558 16218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:29.732702 16218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "db6eb9c4007d461e81b9a51df542e26b"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "7ec9a80745285083156a84877ce75a45"
server_key_iv: "8a300effb18ae285284cfd7e142848f4"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.732826 16218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:29.746002 16218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:29.749475 16218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:29.749692 16218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:29.750329 16218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:29.751192 16218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:29.751261 16218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.751334 16218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:29.751381 16218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.760740 16218 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:32999
I20260928 16:48:29.760761 16347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:32999 every 8 connection(s)
I20260928 16:48:29.761713 16218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:29.762753 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16218
I20260928 16:48:29.762959 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:29.763305 31504 external_mini_cluster.cc:1428] Setting key 54e3822d6f027aa93f40aead56cd706f
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:48:29.776862 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.765268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:37511 (local address 127.30.196.62:39747)
0928 16:48:29.765396 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.765399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.765417 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:48:29.766696 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.766698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.766744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.766872 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.768238 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.768854 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.770248 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.770495 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.770597 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.773166 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.773169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.773171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:29.773195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.774655 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.775203 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.775206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.775208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.775315 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.775654 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.775657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.775659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.775878 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.775989 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.776623 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.776733 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:48:29.778005 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.764296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39747 (local address 127.30.196.1:37511)
0928 16:48:29.765159 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.765217 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.766440 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.767091 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.767098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.767484 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:29.768063 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.768078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.769004 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.769008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.770129 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.770139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.770653 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.770658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.770827 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.771431 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:29.771448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:29.772943 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:29.774787 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:29.774792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:29.774808 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:29.775073 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:29.775411 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:29.775414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:29.775416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:29.775556 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:29.776013 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:29.776019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:29.776333 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.776541 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.776933 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":142,"threads_started":1}
I20260928 16:48:29.779359 16348 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39747
I20260928 16:48:29.779609 16348 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:29.780187 16348 heartbeater.cc:511] Master 127.30.196.62:39747 requested a full tablet report, sending...
I20260928 16:48:29.781896 16155 ts_manager.cc:194] Registered new tserver with Master: db6eb9c4007d461e81b9a51df542e26b (127.30.196.1:32999)
I20260928 16:48:29.783092 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:37511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:48:29.792882 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:29.905341 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:29.905616 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:29.905663 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:29.909775 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:29.909842 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:29.909909 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:29.915439 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:29.916597 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:29.917567 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:29.927549 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.927544 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.927634 16353 server_base.cc:1034] running on GCE node
W20260928 16:48:29.927585 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.928331 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:29.929224 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:29.930413 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790614109930391 us; error 31 us; skew 500 ppm
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:29.933727 16353 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:48:29.934634 16353 webserver.cc:460] Webserver started at http://127.30.196.2:33453/ using document root <none> and password file <none>
I20260928 16:48:29.935205 16353 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:29.935266 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:29.935434 16353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:29.937861 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "96ba54aabbe84a93a2de263487c1d120"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "c55392897516a652d275b0d77128eb18"
server_key_iv: "c1591a69181e11a8a1840d976d8532d5"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.938407 16353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "96ba54aabbe84a93a2de263487c1d120"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "c55392897516a652d275b0d77128eb18"
server_key_iv: "c1591a69181e11a8a1840d976d8532d5"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.942574 16353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:29.945209 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.946506 16353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:29.946651 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "96ba54aabbe84a93a2de263487c1d120"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-5rg1"
server_key: "c55392897516a652d275b0d77128eb18"
server_key_iv: "c1591a69181e11a8a1840d976d8532d5"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.946769 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:29.962637 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:29.966076 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:29.966285 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:29.966889 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:29.968096 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:29.968163 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.968251 16353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:29.968283 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.977607 16353 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:38765
I20260928 16:48:29.977634 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:38765 every 8 connection(s)
I20260928 16:48:29.978520 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:29.979840 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16353
I20260928 16:48:29.979959 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:29.980314 31504 external_mini_cluster.cc:1428] Setting key ef79b8a35f3c8c78f85f9afd5b02c132
Sep 28 16:48:29 dist-test-slave-5rg1 krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614109, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:48:29.993448 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.980410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:35611 (local address 127.30.196.62:39747)
0928 16:48:29.980559 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.980564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.981743 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 16:48:29.982644 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.982651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.982715 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.982855 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.984321 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.984880 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.986072 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.986310 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.986421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.989562 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.989565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.989567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:29.989593 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.991105 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.991766 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.991773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.991775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.991859 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:29.992226 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:29.992229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:29.992231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:29.992433 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.992561 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.993132 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.993272 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:48:29.994304 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.980842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39747 (local address 127.30.196.2:35611)
0928 16:48:29.981589 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.981642 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.982437 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.983133 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.983150 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.983540 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:29.984155 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.984171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.985018 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.985024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.985931 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.985941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.986479 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.986493 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.986767 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.987539 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:29.987555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:29.989369 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:29.991238 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:29.991245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:29.991262 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:29.991614 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:29.991954 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:29.991958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:29.991960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:29.992119 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:29.992569 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:29.992579 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:29.992880 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.993075 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.993422 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":144,"threads_started":1}
I20260928 16:48:29.995551 16483 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39747
I20260928 16:48:29.995838 16483 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:29.996374 16483 heartbeater.cc:511] Master 127.30.196.62:39747 requested a full tablet report, sending...
I20260928 16:48:29.997433 16155 ts_manager.cc:194] Registered new tserver with Master: 96ba54aabbe84a93a2de263487c1d120 (127.30.196.2:38765)
I20260928 16:48:29.998015 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:35611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:48:30.011991 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:30.126456 16488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:30.126718 16488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:30.126763 16488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:30.131165 16488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:30.131240 16488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:30.131318 16488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:30.137980 16488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:39747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:30.139210 16488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:30.140309 16488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:30.150422 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:30.150422 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:30.150422 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:30.150743 16488 server_base.cc:1034] running on GCE node
I20260928 16:48:30.151278 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:30.152122 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:30.153299 16488 hybrid_clock.cc:648] HybridClock initialized: now 1790614110153286 us; error 26 us; skew 500 ppm
Sep 28 16:48:30 dist-test-slave-5rg1 krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614110, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:30.156323 16488 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:48:30.157208 16488 webserver.cc:460] Webserver started at http://127.30.196.3:45071/ using document root <none> and password file <none>
I20260928 16:48:30.157821 16488 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:30.157974 16488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:30.158202 16488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:30.160323 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "1f902dda373625decb49e18bf813c26e"
server_key_iv: "93bf6a2dc688724022ef655d7424fc70"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.160893 16488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "1f902dda373625decb49e18bf813c26e"
server_key_iv: "93bf6a2dc688724022ef655d7424fc70"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.164618 16488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:48:30.167788 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.169296 16488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:48:30.169394 16488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "1f902dda373625decb49e18bf813c26e"
server_key_iv: "93bf6a2dc688724022ef655d7424fc70"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.169482 16488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:30.188277 16488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:30.191639 16488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:30.191861 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:30.192418 16488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:30.193292 16488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:30.193348 16488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.193388 16488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:30.193404 16488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.201846 16488 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:43775
I20260928 16:48:30.201910 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:43775 every 8 connection(s)
I20260928 16:48:30.202736 16488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:30.209704 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16488
I20260928 16:48:30.209808 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:30.210155 31504 external_mini_cluster.cc:1428] Setting key 35ba07f01d1c0ff4e163cba1d239e844
Sep 28 16:48:30 dist-test-slave-5rg1 krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614110, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:30.217247 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.204619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:33703 (local address 127.30.196.62:39747)
0928 16:48:30.204780 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.204784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.205839 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 16:48:30.206733 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.206736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.206785 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.206971 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:30.208234 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.208783 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.209974 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.210224 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.210347 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.213175 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:30.213178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:30.213181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:30.213209 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:30.215068 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:30.215602 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:30.215605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:30.215606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:30.215744 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:30.216097 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:30.216100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:30.216101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:30.216330 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:30.216439 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.217023 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.217127 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:48:30.218230 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.204978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:39747 (local address 127.30.196.3:33703)
0928 16:48:30.205704 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.205743 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.206523 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.207139 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.207145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.207514 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:30.208074 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.208088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.208913 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.208916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.209836 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.209851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.210411 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.210416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.210579 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.211275 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:30.211291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:30.213019 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:30.215172 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:30.215176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:30.215188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:30.215505 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:30.215846 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:30.215852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:30.215857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:30.216010 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:30.216438 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:30.216442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:30.216684 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.216917 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.217194 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":139,"threads_started":1}
I20260928 16:48:30.219581 16618 heartbeater.cc:347] Connected to a master server at 127.30.196.62:39747
I20260928 16:48:30.219861 16618 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:30.220399 16618 heartbeater.cc:511] Master 127.30.196.62:39747 requested a full tablet report, sending...
I20260928 16:48:30.221428 16155 ts_manager.cc:194] Registered new tserver with Master: d79ae1bf8f604e3884bb0c8a8e7f5c98 (127.30.196.3:43775)
I20260928 16:48:30.222018 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:33703
I20260928 16:48:30.224287 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:30.234230 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.226218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.30.196.62:39747)
0928 16:48:30.226375 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.226378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.226541 (+ 163us) server_negotiation.cc:407] Connection header received
0928 16:48:30.226632 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.226635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.226676 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.226749 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:30.227480 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.228145 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.229004 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.229231 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.229326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.230965 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:30.230968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:30.230970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:30.231000 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:30.232482 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:30.233014 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:30.233017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:30.233018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:30.233099 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:30.233337 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:30.233339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:30.233341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:30.233558 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:30.233664 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.233984 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.234105 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:48:30.237576 16155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:30.240631 16155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:30.255489 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.250066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38765 (local address 127.0.0.1:51042)
0928 16:48:30.250319 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.250333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.250427 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.251454 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.251457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.251476 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:30.251580 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.251587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.252563 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.252569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.253988 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.253999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.254908 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.254912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.255019 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.255061 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.255193 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.255325 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":21,"thread_start_us":125,"threads_started":1}
I20260928 16:48:30.255502 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.250545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.30.196.2:38765)
0928 16:48:30.251008 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.251019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.251039 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:30.251119 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.251124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.251299 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.251476 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:30.251698 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.252434 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.254125 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.254764 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.254887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.255041 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.255226 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.255325 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":129,"threads_started":1}
I20260928 16:48:30.255889 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.249331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:32999 (local address 127.0.0.1:59434)
0928 16:48:30.249832 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.249880 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.250017 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.250965 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.250971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.251046 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:30.251225 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.251235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.253061 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.253064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.254153 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.254162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.255283 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.255286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.255378 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.255401 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.255455 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.255526 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":33,"thread_start_us":76,"threads_started":1}
I20260928 16:48:30.256091 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.249532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:43775 (local address 127.0.0.1:43656)
0928 16:48:30.250073 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.250089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.250182 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.251269 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.251273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.251296 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:30.251394 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.251400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.252978 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.252981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.254239 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.254248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.255466 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.255469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.255549 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.255569 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.255876 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.255963 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":74,"thread_start_us":115,"threads_started":1}
I20260928 16:48:30.256558 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.249535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.30.196.1:32999)
0928 16:48:30.250418 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.250435 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.250460 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:48:30.250540 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.250545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.250763 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.251390 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:30.251571 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.252864 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.254290 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.255135 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.256069 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.256226 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.256292 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.256401 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":101,"threads_started":1}
I20260928 16:48:30.256597 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.250173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.30.196.3:43775)
0928 16:48:30.250781 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.250793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.250815 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:48:30.250893 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.250901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.251077 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.251533 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:30.251721 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.252832 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.254362 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.255135 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.255256 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.255404 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.256370 (+ 966us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.256463 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":139,"threads_started":1}
I20260928 16:48:30.258561 16416 tablet_service.cc:1467] Processing CreateTablet for tablet a9c4ef5c364f41b0bcecba352752473d (DEFAULT_TABLE table=test-table [id=86e50687ed6f4f3fa4bc1aac84e692ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:30.259545 16416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c4ef5c364f41b0bcecba352752473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:30.259266 16551 tablet_service.cc:1467] Processing CreateTablet for tablet a9c4ef5c364f41b0bcecba352752473d (DEFAULT_TABLE table=test-table [id=86e50687ed6f4f3fa4bc1aac84e692ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:30.259266 16281 tablet_service.cc:1467] Processing CreateTablet for tablet a9c4ef5c364f41b0bcecba352752473d (DEFAULT_TABLE table=test-table [id=86e50687ed6f4f3fa4bc1aac84e692ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:30.260311 16281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c4ef5c364f41b0bcecba352752473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:30.260329 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9c4ef5c364f41b0bcecba352752473d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:30.266430 16635 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Bootstrap starting.
I20260928 16:48:30.267355 16636 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Bootstrap starting.
I20260928 16:48:30.267962 16637 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: Bootstrap starting.
I20260928 16:48:30.269490 16635 tablet_bootstrap.cc:654] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:30.270254 16637 tablet_bootstrap.cc:654] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:30.270399 16635 log.cc:824] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:30.270697 16636 tablet_bootstrap.cc:654] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:30.271528 16637 log.cc:824] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:30.272042 16636 log.cc:824] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:30.273307 16635 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: No bootstrap required, opened a new log
I20260928 16:48:30.273545 16635 ts_tablet_manager.cc:1397] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:48:30.273693 16637 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: No bootstrap required, opened a new log
I20260928 16:48:30.273985 16636 tablet_bootstrap.cc:492] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: No bootstrap required, opened a new log
I20260928 16:48:30.274011 16637 ts_tablet_manager.cc:1397] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:48:30.274179 16636 ts_tablet_manager.cc:1397] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:48:30.277256 16636 raft_consensus.cc:348] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.277568 16636 raft_consensus.cc:374] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:30.277627 16636 raft_consensus.cc:729] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79ae1bf8f604e3884bb0c8a8e7f5c98, State: Initialized, Role: FOLLOWER
I20260928 16:48:30.277781 16635 raft_consensus.cc:348] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.278096 16635 raft_consensus.cc:374] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:30.278146 16635 raft_consensus.cc:729] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ba54aabbe84a93a2de263487c1d120, State: Initialized, Role: FOLLOWER
I20260928 16:48:30.278002 16637 raft_consensus.cc:348] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.278132 16636 consensus_queue.cc:260] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.278240 16637 raft_consensus.cc:374] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:30.278286 16637 raft_consensus.cc:729] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6eb9c4007d461e81b9a51df542e26b, State: Initialized, Role: FOLLOWER
I20260928 16:48:30.278728 16637 consensus_queue.cc:260] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.279335 16636 ts_tablet_manager.cc:1428] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:30.279363 16635 consensus_queue.cc:260] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.279732 16637 ts_tablet_manager.cc:1428] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:48:30.279748 16618 heartbeater.cc:503] Master 127.30.196.62:39747 was elected leader, sending a full tablet report...
I20260928 16:48:30.280104 16348 heartbeater.cc:503] Master 127.30.196.62:39747 was elected leader, sending a full tablet report...
I20260928 16:48:30.280303 16635 ts_tablet_manager.cc:1428] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:48:30.280607 16483 heartbeater.cc:503] Master 127.30.196.62:39747 was elected leader, sending a full tablet report...
I20260928 16:48:30.341440 16643 raft_consensus.cc:482] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:30.341668 16643 raft_consensus.cc:504] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.342839 16643 leader_election.cc:288] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6eb9c4007d461e81b9a51df542e26b (127.30.196.1:32999), d79ae1bf8f604e3884bb0c8a8e7f5c98 (127.30.196.3:43775)
I20260928 16:48:30.347326 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.343275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:46709 (local address 127.30.196.1:32999)
0928 16:48:30.343413 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.343418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.343435 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:48:30.343529 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.343532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.343581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.343744 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:30.344224 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.344881 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.346244 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.346755 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.346901 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.346959 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.347131 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.347204 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:48:30.347338 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.343173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:32999 (local address 127.30.196.2:46709)
0928 16:48:30.343279 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.343294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.343422 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.343777 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.343780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.343941 (+ 161us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:30.344084 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.344091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.344998 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.345001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.346092 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.346104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.346891 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.346894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.347002 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.347028 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.347116 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.347189 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"spinlock_wait_cycles":1792}
I20260928 16:48:30.348083 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.343269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:59689 (local address 127.30.196.3:43775)
0928 16:48:30.343406 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.343411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.343868 (+ 457us) server_negotiation.cc:407] Connection header received
0928 16:48:30.344082 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.344086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.344146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.344198 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:30.344672 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.345507 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.346882 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.347443 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.347572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.347634 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.347893 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.347973 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:48:30.348115 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c4ef5c364f41b0bcecba352752473d" candidate_uuid: "96ba54aabbe84a93a2de263487c1d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6eb9c4007d461e81b9a51df542e26b" is_pre_election: true
I20260928 16:48:30.348444 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.343171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:43775 (local address 127.30.196.2:59689)
0928 16:48:30.343777 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.343790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.343949 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.344366 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.344370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.344387 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:30.344500 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.344506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.345631 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.345635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.346754 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.346764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.347571 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.347576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.347719 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.347744 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.348231 (+ 487us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.348302 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":134,"spinlock_wait_cycles":3456,"thread_start_us":141,"threads_started":1}
I20260928 16:48:30.348434 16302 raft_consensus.cc:2454] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ba54aabbe84a93a2de263487c1d120 in term 0.
I20260928 16:48:30.349011 16370 leader_election.cc:302] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ba54aabbe84a93a2de263487c1d120, db6eb9c4007d461e81b9a51df542e26b; no voters:
I20260928 16:48:30.349062 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c4ef5c364f41b0bcecba352752473d" candidate_uuid: "96ba54aabbe84a93a2de263487c1d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" is_pre_election: true
I20260928 16:48:30.349309 16643 raft_consensus.cc:2781] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:30.349349 16571 raft_consensus.cc:2454] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96ba54aabbe84a93a2de263487c1d120 in term 0.
I20260928 16:48:30.349419 16643 raft_consensus.cc:482] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:30.349465 16643 raft_consensus.cc:3037] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:30.350631 16643 raft_consensus.cc:504] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.350965 16643 leader_election.cc:288] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [CANDIDATE]: Term 1 election: Requested vote from peers db6eb9c4007d461e81b9a51df542e26b (127.30.196.1:32999), d79ae1bf8f604e3884bb0c8a8e7f5c98 (127.30.196.3:43775)
I20260928 16:48:30.351362 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c4ef5c364f41b0bcecba352752473d" candidate_uuid: "96ba54aabbe84a93a2de263487c1d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98"
I20260928 16:48:30.351388 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9c4ef5c364f41b0bcecba352752473d" candidate_uuid: "96ba54aabbe84a93a2de263487c1d120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6eb9c4007d461e81b9a51df542e26b"
I20260928 16:48:30.351496 16571 raft_consensus.cc:3037] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:30.351503 16302 raft_consensus.cc:3037] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:30.352789 16302 raft_consensus.cc:2454] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96ba54aabbe84a93a2de263487c1d120 in term 1.
I20260928 16:48:30.352789 16571 raft_consensus.cc:2454] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96ba54aabbe84a93a2de263487c1d120 in term 1.
I20260928 16:48:30.353137 16370 leader_election.cc:302] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 96ba54aabbe84a93a2de263487c1d120, d79ae1bf8f604e3884bb0c8a8e7f5c98, db6eb9c4007d461e81b9a51df542e26b; no voters:
I20260928 16:48:30.353344 16643 raft_consensus.cc:2781] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:30.353533 16643 raft_consensus.cc:686] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 1 LEADER]: Becoming Leader. State: Replica: 96ba54aabbe84a93a2de263487c1d120, State: Running, Role: LEADER
I20260928 16:48:30.353868 16643 consensus_queue.cc:237] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } }
I20260928 16:48:30.357102 16155 catalog_manager.cc:5495] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96ba54aabbe84a93a2de263487c1d120 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "96ba54aabbe84a93a2de263487c1d120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96ba54aabbe84a93a2de263487c1d120" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 38765 } health_report { overall_health: HEALTHY } } }
I20260928 16:48:30.380038 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.376499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.30.196.2:38765)
0928 16:48:30.376624 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.376629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.376644 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:48:30.376743 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.376745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.376801 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.376975 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:30.377182 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.377684 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.378537 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.378754 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.378862 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.379169 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:30.379596 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:30.379772 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.379807 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.379865 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:48:30.385898 16571 raft_consensus.cc:1261] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 1 FOLLOWER]: Refusing update from remote peer 96ba54aabbe84a93a2de263487c1d120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:30.385906 16302 raft_consensus.cc:1261] T a9c4ef5c364f41b0bcecba352752473d P db6eb9c4007d461e81b9a51df542e26b [term 1 FOLLOWER]: Refusing update from remote peer 96ba54aabbe84a93a2de263487c1d120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:30.386674 16643 consensus_queue.cc:1035] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6eb9c4007d461e81b9a51df542e26b" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 32999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:30.386880 16645 consensus_queue.cc:1035] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79ae1bf8f604e3884bb0c8a8e7f5c98" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 43775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:30.394531 16648 mvcc.cc:205] Tried to move back new op lower bound from 7334355396133076992 to 7334355396012367872. Current Snapshot: MvccSnapshot[applied={T|T < 7334355396133076992}]
I20260928 16:48:30.395081 16649 mvcc.cc:205] Tried to move back new op lower bound from 7334355396133076992 to 7334355396012367872. Current Snapshot: MvccSnapshot[applied={T|T < 7334355396133076992}]
I20260928 16:48:30.395888 16650 mvcc.cc:205] Tried to move back new op lower bound from 7334355396133076992 to 7334355396012367872. Current Snapshot: MvccSnapshot[applied={T|T < 7334355396133076992}]
I20260928 16:48:30.403194 16154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:30.403371 16154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:30.405783 16154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f74f12ab9d4854bf40cef5d710846a: Sending DeleteTablet for 3 replicas of tablet a9c4ef5c364f41b0bcecba352752473d
I20260928 16:48:30.406642 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c4ef5c364f41b0bcecba352752473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:43656
I20260928 16:48:30.406661 16281 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c4ef5c364f41b0bcecba352752473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59434
I20260928 16:48:30.406755 16416 tablet_service.cc:1514] Processing DeleteTablet for tablet a9c4ef5c364f41b0bcecba352752473d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:30 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:51042
I20260928 16:48:30.407191 16658 tablet_replica.cc:330] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: stopping tablet replica
I20260928 16:48:30.407507 16658 raft_consensus.cc:2229] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:30.407539 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16218
I20260928 16:48:30.407881 16658 raft_consensus.cc:2258] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:30.409020 16660 tablet_replica.cc:330] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: stopping tablet replica
I20260928 16:48:30.409312 16660 raft_consensus.cc:2229] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:30.409710 16660 raft_consensus.cc:2258] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:30.409778 16658 ts_tablet_manager.cc:1905] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:30.412854 16658 ts_tablet_manager.cc:1918] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:30.412953 16658 log.cc:1196] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/a9c4ef5c364f41b0bcecba352752473d
I20260928 16:48:30.413326 16658 ts_tablet_manager.cc:1939] T a9c4ef5c364f41b0bcecba352752473d P d79ae1bf8f604e3884bb0c8a8e7f5c98: Deleting consensus metadata
I20260928 16:48:30.413690 16660 ts_tablet_manager.cc:1905] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:30.414721 16141 catalog_manager.cc:4841] TS d79ae1bf8f604e3884bb0c8a8e7f5c98 (127.30.196.3:43775): tablet a9c4ef5c364f41b0bcecba352752473d (table test-table [id=86e50687ed6f4f3fa4bc1aac84e692ea]) successfully deleted
I20260928 16:48:30.416388 16660 ts_tablet_manager.cc:1918] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:30.416512 16660 log.cc:1196] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/a9c4ef5c364f41b0bcecba352752473d
I20260928 16:48:30.416846 16660 ts_tablet_manager.cc:1939] T a9c4ef5c364f41b0bcecba352752473d P 96ba54aabbe84a93a2de263487c1d120: Deleting consensus metadata
W20260928 16:48:30.416935 16139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:32999 (error 108)
I20260928 16:48:30.417155 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16353
I20260928 16:48:30.418650 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.418362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:32999 (local address 127.0.0.1:59450)
0928 16:48:30.418499 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.418575 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 16:48:30.418865 16139 catalog_manager.cc:4584] TS db6eb9c4007d461e81b9a51df542e26b (127.30.196.1:32999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9c4ef5c364f41b0bcecba352752473d: Network error: Client connection negotiation failed: client connection to 127.30.196.1:32999: connect: Connection refused (error 111)
I20260928 16:48:30.423609 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16488
I20260928 16:48:30.425873 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.423767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:38765 (local address 127.0.0.1:51056)
0928 16:48:30.425691 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.425762 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1810}
I20260928 16:48:30.430023 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16122
2026-09-28T16:48:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T16:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:30Z Disabled control of system clock
I20260928 16:48:30.462188 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:44223
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:44223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:30.577855 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:30.578168 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:30.578208 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:30.582367 16665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:30.582443 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:30.582459 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:30.582475 16665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:30.582491 16665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:30.587440 16665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:44223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:30.588647 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:30.590073 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:30.598861 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:30.598976 16665 server_base.cc:1034] running on GCE node
W20260928 16:48:30.598872 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:30.598876 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:30.599830 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:30.600992 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:30.602165 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790614110602152 us; error 26 us; skew 500 ppm
I20260928 16:48:30.604125 16665 webserver.cc:460] Webserver started at http://127.30.196.62:33401/ using document root <none> and password file <none>
I20260928 16:48:30.604708 16665 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:30.604770 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:30.605000 16665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:30.607208 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "8d6b8f3083244e78bbbe64ce48456455"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "3db9203abc42d632d8b280fea2f3a80d"
server_key_iv: "c8bb700aba487147208c0b1b999276c1"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.607877 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "8d6b8f3083244e78bbbe64ce48456455"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "3db9203abc42d632d8b280fea2f3a80d"
server_key_iv: "c8bb700aba487147208c0b1b999276c1"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.611747 16665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:48:30.614220 16679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.615442 16665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:48:30.615609 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "8d6b8f3083244e78bbbe64ce48456455"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "3db9203abc42d632d8b280fea2f3a80d"
server_key_iv: "c8bb700aba487147208c0b1b999276c1"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.615804 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:30.632545 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:30.633457 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:30.633714 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:30.642015 16665 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:44223
I20260928 16:48:30.642015 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:44223 every 8 connection(s)
I20260928 16:48:30.643146 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:30.646603 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:30.648370 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16665
I20260928 16:48:30.648510 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:30.648829 31504 external_mini_cluster.cc:1428] Setting key 17930a109668fc18f298aad488d98227
I20260928 16:48:30.653755 16733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: Bootstrap starting.
I20260928 16:48:30.656292 16733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:30.657101 16733 log.cc:824] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:30.658672 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.650457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.30.196.62:44223)
0928 16:48:30.651157 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.651172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.651208 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:48:30.651982 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.652004 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.652403 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.652739 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:30.653193 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.654229 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.655145 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.655626 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.655876 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.656843 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:30.656848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:30.656861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:30.657316 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:30.657376 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:30.657397 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:30.657535 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.657688 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.657954 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260928 16:48:30.659217 16733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: No bootstrap required, opened a new log
I20260928 16:48:30.663193 16733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } }
I20260928 16:48:30.663470 16733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:30.663537 16733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6b8f3083244e78bbbe64ce48456455, State: Initialized, Role: FOLLOWER
I20260928 16:48:30.664027 16733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } }
I20260928 16:48:30.664182 16733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:30.664268 16733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:30.664367 16733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:30.665356 16733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } }
I20260928 16:48:30.665707 16733 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6b8f3083244e78bbbe64ce48456455; no voters:
I20260928 16:48:30.666035 16733 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:30.666200 16738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:30.666445 16738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6b8f3083244e78bbbe64ce48456455, State: Running, Role: LEADER
I20260928 16:48:30.666733 16738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } }
I20260928 16:48:30.667330 16733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:30.668572 16740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b8f3083244e78bbbe64ce48456455. Latest consensus state: current_term: 1 leader_uuid: "8d6b8f3083244e78bbbe64ce48456455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } } }
I20260928 16:48:30.668572 16739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6b8f3083244e78bbbe64ce48456455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6b8f3083244e78bbbe64ce48456455" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 44223 } } }
I20260928 16:48:30.668752 16739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:30.668752 16740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:30.669521 16749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:30.672488 16749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:30.678054 16749 catalog_manager.cc:1347] Generated new cluster ID: ad2fdd5bb90548099425aef996de9d2f
I20260928 16:48:30.678172 16749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:30.707950 16749 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:30.709151 16749 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:30.713521 16749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: Generated new TSK 0
I20260928 16:48:30.714303 16749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:30.720851 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--builtin_ntp_servers=127.30.196.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:30.843462 16757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:30.843776 16757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:30.843843 16757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:30.848335 16757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:30.848395 16757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:30.848490 16757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:30.853377 16757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:30.854513 16757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:30.855985 16757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:30.865863 16765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:30.865855 16762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:30.865872 16763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:30.866111 16757 server_base.cc:1034] running on GCE node
I20260928 16:48:30.866663 16757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:30.867296 16757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:30.868458 16757 hybrid_clock.cc:648] HybridClock initialized: now 1790614110868437 us; error 32 us; skew 500 ppm
I20260928 16:48:30.870580 16757 webserver.cc:460] Webserver started at http://127.30.196.1:35035/ using document root <none> and password file <none>
I20260928 16:48:30.871133 16757 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:30.871191 16757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:30.871356 16757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:30.873454 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "8d139389ce5b4852a96de45c39050531"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "b644532d6ebd6b640d0e0a245898021c"
server_key_iv: "1ec6b7f2459fdc1aa00659188f0df81d"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.873984 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "8d139389ce5b4852a96de45c39050531"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "b644532d6ebd6b640d0e0a245898021c"
server_key_iv: "1ec6b7f2459fdc1aa00659188f0df81d"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.877677 16757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:48:30.880172 16771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.881438 16757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:30.881587 16757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "8d139389ce5b4852a96de45c39050531"
format_stamp: "Formatted at 2026-09-28 16:48:30 on dist-test-slave-5rg1"
server_key: "b644532d6ebd6b640d0e0a245898021c"
server_key_iv: "1ec6b7f2459fdc1aa00659188f0df81d"
server_key_version: "encryptionkey@0"
I20260928 16:48:30.881704 16757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:30.896667 16757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:30.897428 16757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:30.897645 16757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:30.898284 16757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:30.899296 16757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:30.899365 16757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.899438 16757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:30.899478 16757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:30.908751 16757 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:33699
I20260928 16:48:30.908818 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:33699 every 8 connection(s)
I20260928 16:48:30.909811 16757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:30.918684 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16757
I20260928 16:48:30.918808 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:30.919137 31504 external_mini_cluster.cc:1428] Setting key 9c6e79074497414e2724200e72b22836
I20260928 16:48:30.919848 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.911785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:54271 (local address 127.30.196.62:44223)
0928 16:48:30.911951 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.911961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.913099 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 16:48:30.914550 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.914554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.914620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.914720 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:30.915659 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.916258 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.917439 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.917688 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.917803 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.919213 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:30.919218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:30.919220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:30.919237 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:30.919265 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:30.919274 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:30.919368 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.919575 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.919653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:48:30.920892 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.912165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:44223 (local address 127.30.196.1:54271)
0928 16:48:30.912933 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.912977 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:30.914327 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:30.914907 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:30.914913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:30.915001 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:30.915505 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.915518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.916403 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.916406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.917305 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:30.917317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.917865 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.917869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:30.918033 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.918742 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:30.918769 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:30.918902 (+ 133us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:30.919013 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:30.919376 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:30.919387 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:30.919393 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:48:30.919538 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 16:48:30.919912 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":105,"threads_started":1}
I20260928 16:48:30.921368 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--builtin_ntp_servers=127.30.196.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:30.923380 16885 heartbeater.cc:347] Connected to a master server at 127.30.196.62:44223
I20260928 16:48:30.923645 16885 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:30.924299 16885 heartbeater.cc:511] Master 127.30.196.62:44223 requested a full tablet report, sending...
I20260928 16:48:30.926131 16690 ts_manager.cc:194] Registered new tserver with Master: 8d139389ce5b4852a96de45c39050531 (127.30.196.1:33699)
I20260928 16:48:30.928038 16690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.1:54271
W20260928 16:48:31.036371 16888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:31.036665 16888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:31.036705 16888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:31.040841 16888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:31.040890 16888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:31.040947 16888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:31.045643 16888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:31.046643 16888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:31.047937 16888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:31.057526 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:31.057531 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:31.057605 16888 server_base.cc:1034] running on GCE node
W20260928 16:48:31.057526 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:31.058190 16888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:31.058791 16888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:31.059942 16888 hybrid_clock.cc:648] HybridClock initialized: now 1790614111059912 us; error 44 us; skew 500 ppm
I20260928 16:48:31.061839 16888 webserver.cc:460] Webserver started at http://127.30.196.2:42929/ using document root <none> and password file <none>
I20260928 16:48:31.062428 16888 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:31.062491 16888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:31.062700 16888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:31.064752 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "6c19c15135f14c08bac6286242cfe1b6"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "4d11e0702bdefe07ff6c978b8bffd605"
server_key_iv: "f1cb651b538bb1b90599e0796999fcf8"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.065292 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "6c19c15135f14c08bac6286242cfe1b6"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "4d11e0702bdefe07ff6c978b8bffd605"
server_key_iv: "f1cb651b538bb1b90599e0796999fcf8"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.069062 16888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:48:31.071395 16902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.072589 16888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:31.072705 16888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "6c19c15135f14c08bac6286242cfe1b6"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "4d11e0702bdefe07ff6c978b8bffd605"
server_key_iv: "f1cb651b538bb1b90599e0796999fcf8"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.072806 16888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:31.087996 16888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:31.088682 16888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:31.088883 16888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:31.089473 16888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:31.090405 16888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:31.090451 16888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.090492 16888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:31.090525 16888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.099311 16888 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:37377
I20260928 16:48:31.099399 17018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:37377 every 8 connection(s)
I20260928 16:48:31.100306 16888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:31.109540 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 16888
I20260928 16:48:31.109694 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:31.110129 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.102213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:34609 (local address 127.30.196.62:44223)
0928 16:48:31.102386 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.102400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.103602 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 16:48:31.105284 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.105287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.105345 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.105428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:31.106408 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.106957 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.108104 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.108323 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.108414 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.109523 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:31.109528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:31.109531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:31.109548 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:31.109574 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:31.109585 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:31.109676 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.109885 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.109965 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:31.110155 31504 external_mini_cluster.cc:1428] Setting key 673bca5a01f4d42dd546bda1a1d5fc2f
I20260928 16:48:31.111078 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.102640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:44223 (local address 127.30.196.2:34609)
0928 16:48:31.103438 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.103483 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.105051 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.105624 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.105631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.105724 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:31.106246 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.106260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.107092 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.107094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.107992 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.108002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.108474 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.108477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.108619 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.109244 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:31.109246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:31.109285 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:31.109353 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:31.109708 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:31.109718 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:31.109724 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.109858 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.110094 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
I20260928 16:48:31.112246 17019 heartbeater.cc:347] Connected to a master server at 127.30.196.62:44223
I20260928 16:48:31.112401 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--builtin_ntp_servers=127.30.196.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:31.112510 17019 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:31.113036 17019 heartbeater.cc:511] Master 127.30.196.62:44223 requested a full tablet report, sending...
I20260928 16:48:31.114099 16690 ts_manager.cc:194] Registered new tserver with Master: 6c19c15135f14c08bac6286242cfe1b6 (127.30.196.2:37377)
I20260928 16:48:31.114926 16690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.2:34609
W20260928 16:48:31.230321 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:31.230602 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:31.230643 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:31.235039 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:31.235118 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:31.235177 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:31.240222 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:31.241520 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:31.242985 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:31.253540 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:31.253545 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:31.253544 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:31.254014 17022 server_base.cc:1034] running on GCE node
I20260928 16:48:31.254514 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:31.255178 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:31.256349 17022 hybrid_clock.cc:648] HybridClock initialized: now 1790614111256320 us; error 48 us; skew 500 ppm
I20260928 16:48:31.258426 17022 webserver.cc:460] Webserver started at http://127.30.196.3:37255/ using document root <none> and password file <none>
I20260928 16:48:31.259025 17022 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:31.259114 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:31.259322 17022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:31.261459 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "1ae380e62ec546fd8bfda2bf6d8a356b"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "ecb864ed805f2867986543c4401cd043"
server_key_iv: "bf407816a7432d66fcd2e0ff45aec3a4"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.262069 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "1ae380e62ec546fd8bfda2bf6d8a356b"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "ecb864ed805f2867986543c4401cd043"
server_key_iv: "bf407816a7432d66fcd2e0ff45aec3a4"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.265846 17022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 16:48:31.268415 17036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.269780 17022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:48:31.269918 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "1ae380e62ec546fd8bfda2bf6d8a356b"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-5rg1"
server_key: "ecb864ed805f2867986543c4401cd043"
server_key_iv: "bf407816a7432d66fcd2e0ff45aec3a4"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.270032 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:31.284432 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:31.285384 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:31.285595 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:31.286227 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:31.287285 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:31.287358 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.287436 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:31.287477 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.296633 17022 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:41153
I20260928 16:48:31.296656 17150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:41153 every 8 connection(s)
I20260928 16:48:31.297595 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:31.300004 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17022
I20260928 16:48:31.300136 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:31.300455 31504 external_mini_cluster.cc:1428] Setting key c6924ec7aa75024db24f69ee6a36fa69
I20260928 16:48:31.308295 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.299759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:44555 (local address 127.30.196.62:44223)
0928 16:48:31.299908 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.299912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.301354 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 16:48:31.303042 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.303044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.303092 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.303160 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:31.304220 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.304762 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.306024 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.306276 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.306379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.307726 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:31.307732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:31.307740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:31.307755 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:31.307782 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:31.307791 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:31.307938 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.308106 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.308168 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:48:31.309260 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.300212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:44223 (local address 127.30.196.3:44555)
0928 16:48:31.301174 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.301233 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.302843 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.303330 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.303336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.303454 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:31.304049 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.304063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.304930 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.304943 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.305864 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.305876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.306463 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.306466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.306622 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.307327 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:31.307329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:31.307372 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:31.307468 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:31.307941 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:31.307951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:31.307956 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.308086 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.308329 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":152,"threads_started":1}
I20260928 16:48:31.310653 17151 heartbeater.cc:347] Connected to a master server at 127.30.196.62:44223
I20260928 16:48:31.310969 17151 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:31.311553 17151 heartbeater.cc:511] Master 127.30.196.62:44223 requested a full tablet report, sending...
I20260928 16:48:31.312695 16690 ts_manager.cc:194] Registered new tserver with Master: 1ae380e62ec546fd8bfda2bf6d8a356b (127.30.196.3:41153)
I20260928 16:48:31.313313 16690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.3:44555
I20260928 16:48:31.314726 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:31.320878 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.316885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.30.196.62:44223)
0928 16:48:31.317004 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.317008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.317211 (+ 203us) server_negotiation.cc:407] Connection header received
0928 16:48:31.317307 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.317309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.317348 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.317439 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:31.317683 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.318220 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.319262 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.319502 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.319585 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.320556 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:31.320559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:31.320561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:31.320573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:31.320592 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:31.320600 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:31.320665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.320721 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.320770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:48:31.323999 16696 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:31.327003 16696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:31.342381 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.337036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.30.196.3:41153)
0928 16:48:31.338177 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.338188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.338209 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:31.338276 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.338280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.338468 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.338616 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:31.338811 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339794 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.341154 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.341673 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.341800 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.342017 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.342105 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.342205 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":172,"threads_started":1}
I20260928 16:48:31.342376 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.336815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41153 (local address 127.0.0.1:45220)
0928 16:48:31.337853 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.337867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.337940 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.338591 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.338593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.338611 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:31.338696 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.338702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339922 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.339926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.341030 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.341038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.341802 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.341805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.341913 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.341953 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.342073 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.342216 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":25,"thread_start_us":92,"threads_started":1}
I20260928 16:48:31.342682 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.336186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33699 (local address 127.0.0.1:57166)
0928 16:48:31.337176 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.337190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.337274 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.338197 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.338201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.338220 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:31.338325 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.338331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339692 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.339696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.341046 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.341058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.342093 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.342097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.342276 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.342390 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.342458 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.342546 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":129,"thread_start_us":98,"threads_started":1}
I20260928 16:48:31.342798 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.336205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.30.196.2:37377)
0928 16:48:31.336586 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.336599 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.337136 (+ 537us) server_negotiation.cc:407] Connection header received
0928 16:48:31.337229 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.337236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.337448 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.337608 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:31.337915 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339126 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.340843 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.341701 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.341814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.342030 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.342531 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.342633 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":189,"threads_started":1}
I20260928 16:48:31.343176 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.336069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37377 (local address 127.0.0.1:34300)
0928 16:48:31.336791 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.336845 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.336984 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.337597 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.337603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.337625 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:31.337783 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.337791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339243 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.339246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.340712 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.340723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.342093 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.342097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.342279 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.342392 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.342937 (+ 545us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.343029 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":139,"thread_start_us":90,"threads_started":1}
I20260928 16:48:31.343487 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.337222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.30.196.1:33699)
0928 16:48:31.337726 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.337739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.337764 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:48:31.337842 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.337863 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.338057 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.338204 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:31.338442 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.339532 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.341191 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.341964 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.342953 (+ 989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.343153 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.343205 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.343314 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":114,"threads_started":1}
I20260928 16:48:31.345767 16952 tablet_service.cc:1467] Processing CreateTablet for tablet b29193ed7a704bd789157ed9052f0117 (DEFAULT_TABLE table=test-table [id=8114ec913b0047eea3694ced1a576dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:31.345770 17085 tablet_service.cc:1467] Processing CreateTablet for tablet b29193ed7a704bd789157ed9052f0117 (DEFAULT_TABLE table=test-table [id=8114ec913b0047eea3694ced1a576dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:31.346768 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29193ed7a704bd789157ed9052f0117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:31.346777 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29193ed7a704bd789157ed9052f0117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:31.348356 16819 tablet_service.cc:1467] Processing CreateTablet for tablet b29193ed7a704bd789157ed9052f0117 (DEFAULT_TABLE table=test-table [id=8114ec913b0047eea3694ced1a576dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:31.349635 16819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29193ed7a704bd789157ed9052f0117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:31.353896 17168 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Bootstrap starting.
I20260928 16:48:31.355346 17169 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Bootstrap starting.
I20260928 16:48:31.356427 17168 tablet_bootstrap.cc:654] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:31.357234 17168 log.cc:824] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:31.357615 17169 tablet_bootstrap.cc:654] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:31.358042 17170 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Bootstrap starting.
I20260928 16:48:31.358400 17169 log.cc:824] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:31.359619 17168 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: No bootstrap required, opened a new log
I20260928 16:48:31.359921 17168 ts_tablet_manager.cc:1397] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260928 16:48:31.360435 17169 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: No bootstrap required, opened a new log
I20260928 16:48:31.360637 17169 ts_tablet_manager.cc:1397] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:31.361285 17170 tablet_bootstrap.cc:654] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:31.362126 17170 log.cc:824] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:31.364169 17169 raft_consensus.cc:348] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.364419 17169 raft_consensus.cc:374] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:31.364485 17169 raft_consensus.cc:729] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae380e62ec546fd8bfda2bf6d8a356b, State: Initialized, Role: FOLLOWER
I20260928 16:48:31.364347 17168 raft_consensus.cc:348] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.364660 17168 raft_consensus.cc:374] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:31.364729 17168 raft_consensus.cc:729] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c19c15135f14c08bac6286242cfe1b6, State: Initialized, Role: FOLLOWER
I20260928 16:48:31.365033 17170 tablet_bootstrap.cc:492] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: No bootstrap required, opened a new log
I20260928 16:48:31.364991 17169 consensus_queue.cc:260] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.365243 17170 ts_tablet_manager.cc:1397] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 16:48:31.365190 17168 consensus_queue.cc:260] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.366081 17169 ts_tablet_manager.cc:1428] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:31.366144 17168 ts_tablet_manager.cc:1428] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260928 16:48:31.366442 17151 heartbeater.cc:503] Master 127.30.196.62:44223 was elected leader, sending a full tablet report...
I20260928 16:48:31.366474 17019 heartbeater.cc:503] Master 127.30.196.62:44223 was elected leader, sending a full tablet report...
I20260928 16:48:31.369571 17170 raft_consensus.cc:348] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.369866 17170 raft_consensus.cc:374] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:31.369936 17170 raft_consensus.cc:729] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d139389ce5b4852a96de45c39050531, State: Initialized, Role: FOLLOWER
I20260928 16:48:31.370512 17170 consensus_queue.cc:260] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.371383 16885 heartbeater.cc:503] Master 127.30.196.62:44223 was elected leader, sending a full tablet report...
I20260928 16:48:31.371599 17170 ts_tablet_manager.cc:1428] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
W20260928 16:48:31.411074 16886 tablet.cc:2369] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:31.548651 17152 tablet.cc:2369] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:31.601595 17020 tablet.cc:2369] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:31.658742 17174 raft_consensus.cc:482] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:31.659026 17174 raft_consensus.cc:504] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.660365 17174 leader_election.cc:288] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c19c15135f14c08bac6286242cfe1b6 (127.30.196.2:37377), 8d139389ce5b4852a96de45c39050531 (127.30.196.1:33699)
I20260928 16:48:31.664389 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.660854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:59977 (local address 127.30.196.2:37377)
0928 16:48:31.661068 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.661072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.661084 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:48:31.661130 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.661132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.661173 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.661265 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:31.661542 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.662166 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.663362 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.663797 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.663930 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.663985 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.664174 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.664246 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 16:48:31.664389 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.660581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:37377 (local address 127.30.196.3:59977)
0928 16:48:31.660734 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.660750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.660856 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.661276 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.661279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.661297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:31.661425 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.661431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.662302 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.662306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.663214 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.663225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.663925 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.663929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.664052 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.664071 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.664157 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.664238 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":26}
I20260928 16:48:31.664836 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.660696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33699 (local address 127.30.196.3:50427)
0928 16:48:31.661110 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.661122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:31.661241 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:31.661520 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:31.661522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:31.661538 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:31.661674 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.661680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.662440 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.662444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.663640 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:31.663651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.664353 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.664356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:31.664466 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.664497 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:48:31.664624 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:48:31.664689 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":22,"thread_start_us":205,"threads_started":1}
I20260928 16:48:31.664903 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.660872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:50427 (local address 127.30.196.1:33699)
0928 16:48:31.661021 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.661026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.661199 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:48:31.661368 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.661370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.661415 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.661507 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:31.661787 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.662317 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.663831 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.664221 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.664352 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.664401 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.664661 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.664754 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:31.665159 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29193ed7a704bd789157ed9052f0117" candidate_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c19c15135f14c08bac6286242cfe1b6" is_pre_election: true
I20260928 16:48:31.665483 16972 raft_consensus.cc:2454] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae380e62ec546fd8bfda2bf6d8a356b in term 0.
I20260928 16:48:31.665783 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29193ed7a704bd789157ed9052f0117" candidate_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d139389ce5b4852a96de45c39050531" is_pre_election: true
I20260928 16:48:31.666078 16839 raft_consensus.cc:2454] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae380e62ec546fd8bfda2bf6d8a356b in term 0.
I20260928 16:48:31.666198 17038 leader_election.cc:302] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae380e62ec546fd8bfda2bf6d8a356b, 6c19c15135f14c08bac6286242cfe1b6; no voters:
I20260928 16:48:31.666512 17174 raft_consensus.cc:2781] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:31.666602 17174 raft_consensus.cc:482] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:31.666646 17174 raft_consensus.cc:3037] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:31.667830 17174 raft_consensus.cc:504] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.668217 17174 leader_election.cc:288] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [CANDIDATE]: Term 1 election: Requested vote from peers 6c19c15135f14c08bac6286242cfe1b6 (127.30.196.2:37377), 8d139389ce5b4852a96de45c39050531 (127.30.196.1:33699)
I20260928 16:48:31.668582 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29193ed7a704bd789157ed9052f0117" candidate_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c19c15135f14c08bac6286242cfe1b6"
I20260928 16:48:31.668676 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29193ed7a704bd789157ed9052f0117" candidate_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d139389ce5b4852a96de45c39050531"
I20260928 16:48:31.668757 16972 raft_consensus.cc:3037] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:31.668864 16839 raft_consensus.cc:3037] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:31.669901 16972 raft_consensus.cc:2454] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae380e62ec546fd8bfda2bf6d8a356b in term 1.
I20260928 16:48:31.670231 16839 raft_consensus.cc:2454] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae380e62ec546fd8bfda2bf6d8a356b in term 1.
I20260928 16:48:31.670257 17038 leader_election.cc:302] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae380e62ec546fd8bfda2bf6d8a356b, 6c19c15135f14c08bac6286242cfe1b6; no voters:
I20260928 16:48:31.670418 17174 raft_consensus.cc:2781] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:31.670662 17174 raft_consensus.cc:686] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 1 LEADER]: Becoming Leader. State: Replica: 1ae380e62ec546fd8bfda2bf6d8a356b, State: Running, Role: LEADER
I20260928 16:48:31.671046 17174 consensus_queue.cc:237] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } }
I20260928 16:48:31.674465 16696 catalog_manager.cc:5495] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae380e62ec546fd8bfda2bf6d8a356b (127.30.196.3). New cstate: current_term: 1 leader_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae380e62ec546fd8bfda2bf6d8a356b" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41153 } health_report { overall_health: HEALTHY } } }
I20260928 16:48:31.708015 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.704669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.30.196.3:41153)
0928 16:48:31.704799 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.704802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.704814 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:48:31.704898 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.704901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.704968 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.705064 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:31.705284 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.705824 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.706645 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.706860 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.706955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:31.707210 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:31.707644 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:31.707803 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:31.707834 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 16:48:31.707882 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:48:31.714213 16972 raft_consensus.cc:1261] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 1 FOLLOWER]: Refusing update from remote peer 1ae380e62ec546fd8bfda2bf6d8a356b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:31.714213 16839 raft_consensus.cc:1261] T b29193ed7a704bd789157ed9052f0117 P 8d139389ce5b4852a96de45c39050531 [term 1 FOLLOWER]: Refusing update from remote peer 1ae380e62ec546fd8bfda2bf6d8a356b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:31.714993 17178 consensus_queue.cc:1035] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c19c15135f14c08bac6286242cfe1b6" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:31.715240 17174 consensus_queue.cc:1035] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d139389ce5b4852a96de45c39050531" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:31.726740 17183 mvcc.cc:205] Tried to move back new op lower bound from 7334355401572864000 to 7334355401407782912. Current Snapshot: MvccSnapshot[applied={T|T < 7334355401572864000}]
I20260928 16:48:31.732445 16690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:31.732620 16690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:31.735266 16690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d6b8f3083244e78bbbe64ce48456455: Sending DeleteTablet for 3 replicas of tablet b29193ed7a704bd789157ed9052f0117
I20260928 16:48:31.736287 16819 tablet_service.cc:1514] Processing DeleteTablet for tablet b29193ed7a704bd789157ed9052f0117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:31 UTC) from {username='slave'} at 127.0.0.1:57166
I20260928 16:48:31.736438 17085 tablet_service.cc:1514] Processing DeleteTablet for tablet b29193ed7a704bd789157ed9052f0117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:31 UTC) from {username='slave'} at 127.0.0.1:45220
I20260928 16:48:31.736424 16952 tablet_service.cc:1514] Processing DeleteTablet for tablet b29193ed7a704bd789157ed9052f0117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:31 UTC) from {username='slave'} at 127.0.0.1:34300
I20260928 16:48:31.737092 17192 tablet_replica.cc:330] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: stopping tablet replica
I20260928 16:48:31.737092 17191 tablet_replica.cc:330] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: stopping tablet replica
I20260928 16:48:31.737445 17192 raft_consensus.cc:2229] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:31.737444 17191 raft_consensus.cc:2229] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:31.737510 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16757
I20260928 16:48:31.737808 17191 raft_consensus.cc:2258] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:31.737954 17192 raft_consensus.cc:2258] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:31.739914 17192 ts_tablet_manager.cc:1905] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:31.742422 17191 ts_tablet_manager.cc:1905] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:31.743212 17192 ts_tablet_manager.cc:1918] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:31.743394 17192 log.cc:1196] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/b29193ed7a704bd789157ed9052f0117
I20260928 16:48:31.743830 17192 ts_tablet_manager.cc:1939] T b29193ed7a704bd789157ed9052f0117 P 1ae380e62ec546fd8bfda2bf6d8a356b: Deleting consensus metadata
I20260928 16:48:31.745304 17191 ts_tablet_manager.cc:1918] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:31.745280 16683 catalog_manager.cc:4841] TS 1ae380e62ec546fd8bfda2bf6d8a356b (127.30.196.3:41153): tablet b29193ed7a704bd789157ed9052f0117 (table test-table [id=8114ec913b0047eea3694ced1a576dac]) successfully deleted
I20260928 16:48:31.745397 17191 log.cc:1196] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/b29193ed7a704bd789157ed9052f0117
I20260928 16:48:31.745728 17191 ts_tablet_manager.cc:1939] T b29193ed7a704bd789157ed9052f0117 P 6c19c15135f14c08bac6286242cfe1b6: Deleting consensus metadata
I20260928 16:48:31.746973 16681 catalog_manager.cc:4841] TS 6c19c15135f14c08bac6286242cfe1b6 (127.30.196.2:37377): tablet b29193ed7a704bd789157ed9052f0117 (table test-table [id=8114ec913b0047eea3694ced1a576dac]) successfully deleted
W20260928 16:48:31.747500 16683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:33699 (error 108)
I20260928 16:48:31.747790 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16888
I20260928 16:48:31.750707 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.750320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:33699 (local address 127.0.0.1:57174)
0928 16:48:31.750504 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:31.750594 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:48:31.750905 16683 catalog_manager.cc:4584] TS 8d139389ce5b4852a96de45c39050531 (127.30.196.1:33699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b29193ed7a704bd789157ed9052f0117: Network error: Client connection negotiation failed: client connection to 127.30.196.1:33699: connect: Connection refused (error 111)
I20260928 16:48:31.754738 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17022
I20260928 16:48:31.760911 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 16665
2026-09-28T16:48:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:31 dist-test-slave-5rg1 krb5kdc[17196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:31 dist-test-slave-5rg1 krb5kdc[17196](info): set up 2 sockets
Sep 28 16:48:31 dist-test-slave-5rg1 krb5kdc[17196](info): commencing operation
krb5kdc: starting...
W20260928 16:48:32.824571 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:32 dist-test-slave-5rg1 krb5kdc[17196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:48:32.954908 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34561
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.068306 17210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.068604 17210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.068682 17210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.072836 17210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:33.072922 17210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.072969 17210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.073020 17210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:33.073056 17210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:33.078370 17210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:33.079475 17210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.080653 17210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.089442 17216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.089455 17218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.089455 17215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.089636 17210 server_base.cc:1034] running on GCE node
I20260928 16:48:33.090299 17210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.091486 17210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.092669 17210 hybrid_clock.cc:648] HybridClock initialized: now 1790614113092658 us; error 61 us; skew 500 ppm
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.095489 17210 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:33.096436 17210 webserver.cc:460] Webserver started at http://127.30.196.62:44303/ using document root <none> and password file <none>
I20260928 16:48:33.097003 17210 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.097090 17210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.097311 17210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.099375 17210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "d8cf4f498aa44c5c8e34029f15aff254"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "c45c8a558dce395b9f97ae98395ad3e9"
server_key_iv: "03d76b94f994020e1e6adc5e047f5ec4"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.099984 17210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "d8cf4f498aa44c5c8e34029f15aff254"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "c45c8a558dce395b9f97ae98395ad3e9"
server_key_iv: "03d76b94f994020e1e6adc5e047f5ec4"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.104240 17210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:33.106917 17226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.108422 17210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:33.108587 17210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "d8cf4f498aa44c5c8e34029f15aff254"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "c45c8a558dce395b9f97ae98395ad3e9"
server_key_iv: "03d76b94f994020e1e6adc5e047f5ec4"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.108721 17210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.136335 17210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.139801 17210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.140033 17210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.148360 17210 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:34561
I20260928 16:48:33.148401 17278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:34561 every 8 connection(s)
I20260928 16:48:33.149398 17210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:33.152012 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17210
I20260928 16:48:33.152165 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:33.152490 31504 external_mini_cluster.cc:1428] Setting key ee76a07fa7e41371b5bd84b21370f9c3
I20260928 16:48:33.152611 17279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.160600 17279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: Bootstrap starting.
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:33.163515 17279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.164606 17279 log.cc:824] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.167289 17279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: No bootstrap required, opened a new log
I20260928 16:48:33.169777 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.154090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.30.196.62:34561)
0928 16:48:33.154550 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.154570 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.154616 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:48:33.155590 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.155617 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.156109 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.156531 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.157284 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.158798 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.159905 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.160635 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.160815 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.163887 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.163895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.163913 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.163956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.166229 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.166756 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.166763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.166769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.166919 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.167235 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.167238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.167240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.167821 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.168028 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.168371 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.168694 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260928 16:48:33.172104 17279 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } }
I20260928 16:48:33.172441 17279 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.172520 17279 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cf4f498aa44c5c8e34029f15aff254, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.172943 17279 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } }
I20260928 16:48:33.173101 17279 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:33.173167 17279 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:33.173270 17279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:33.174526 17279 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } }
I20260928 16:48:33.174896 17279 leader_election.cc:302] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8cf4f498aa44c5c8e34029f15aff254; no voters:
I20260928 16:48:33.175240 17279 leader_election.cc:288] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:33.175393 17284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:33.175637 17284 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [term 1 LEADER]: Becoming Leader. State: Replica: d8cf4f498aa44c5c8e34029f15aff254, State: Running, Role: LEADER
I20260928 16:48:33.176076 17284 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } }
I20260928 16:48:33.176550 17279 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:33.177692 17285 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8cf4f498aa44c5c8e34029f15aff254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } } }
I20260928 16:48:33.177920 17285 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:33.178535 17294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:33.180017 17286 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cf4f498aa44c5c8e34029f15aff254. Latest consensus state: current_term: 1 leader_uuid: "d8cf4f498aa44c5c8e34029f15aff254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf4f498aa44c5c8e34029f15aff254" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 34561 } } }
I20260928 16:48:33.180284 17286 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:33.182837 17294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:33.188961 17294 catalog_manager.cc:1347] Generated new cluster ID: 17131ef2f7e94549aea9625a7e002ab8
I20260928 16:48:33.189044 17294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:33.201162 17294 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:33.202261 17294 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:33.211589 17294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: Generated new TSK 0
I20260928 16:48:33.212422 17294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:48:33.251228 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.368768 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.369134 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.369207 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.373437 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.373507 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.373596 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:33.379295 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:33.380520 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.381620 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.392071 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.392040 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.392365 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.392257 17305 server_base.cc:1034] running on GCE node
I20260928 16:48:33.392773 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.393450 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.394642 17305 hybrid_clock.cc:648] HybridClock initialized: now 1790614113394626 us; error 29 us; skew 500 ppm
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.405117 17305 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:48:33.406168 17305 webserver.cc:460] Webserver started at http://127.30.196.1:39163/ using document root <none> and password file <none>
I20260928 16:48:33.406788 17305 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.406886 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.407137 17305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.409328 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "0e42660b5ceb41de8f1f390b399d3c7e"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "9be87ef357a5e09aef8eeeee97b5d584"
server_key_iv: "c8f516a72b9b67744ad6b69e1eb2d170"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.409950 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "0e42660b5ceb41de8f1f390b399d3c7e"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "9be87ef357a5e09aef8eeeee97b5d584"
server_key_iv: "c8f516a72b9b67744ad6b69e1eb2d170"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.414047 17305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:48:33.416658 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.417950 17305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:33.418093 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "0e42660b5ceb41de8f1f390b399d3c7e"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "9be87ef357a5e09aef8eeeee97b5d584"
server_key_iv: "c8f516a72b9b67744ad6b69e1eb2d170"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.418212 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.440548 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.444267 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.444482 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.445174 17305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:33.446076 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:33.446148 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.446219 17305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:33.446264 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.456459 17305 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:40593
I20260928 16:48:33.456486 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:40593 every 8 connection(s)
I20260928 16:48:33.457448 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:33.458673 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17305
I20260928 16:48:33.458804 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:33.459132 31504 external_mini_cluster.cc:1428] Setting key b1c254d97d8fcab0c5a4c4c4bd9fffae
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:48:33.471793 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.459572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:57281 (local address 127.30.196.62:34561)
0928 16:48:33.459749 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.459754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.460943 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 16:48:33.461970 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.461977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.462039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.462134 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.463338 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.463971 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.465093 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.465398 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.465535 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.468121 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.468125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.468132 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.468157 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.469653 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.470181 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.470185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.470186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.470276 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.470623 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.470626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.470628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.470820 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.470954 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.471451 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.471595 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:48:33.472759 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.460048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34561 (local address 127.30.196.1:57281)
0928 16:48:33.460783 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.460845 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.461749 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.462287 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.462293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.462657 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:33.463175 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.463190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.464098 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.464102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.464960 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.464970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.465558 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.465562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.465748 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.466351 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:33.466367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:33.467947 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:33.469774 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.469780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.469795 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.470076 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.470369 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.470372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.470374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.470518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.470952 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:33.470960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:33.471222 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.471412 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.471784 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":151,"threads_started":1}
I20260928 16:48:33.474094 17435 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34561
I20260928 16:48:33.474344 17435 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:33.475112 17435 heartbeater.cc:511] Master 127.30.196.62:34561 requested a full tablet report, sending...
I20260928 16:48:33.476922 17243 ts_manager.cc:194] Registered new tserver with Master: 0e42660b5ceb41de8f1f390b399d3c7e (127.30.196.1:40593)
I20260928 16:48:33.478268 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:57281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:48:33.491035 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.604666 17440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.604993 17440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.605077 17440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.609272 17440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.609355 17440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.609448 17440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:33.615036 17440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:33.616245 17440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.617276 17440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.627074 17445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.627079 17448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.627099 17446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.627369 17440 server_base.cc:1034] running on GCE node
I20260928 16:48:33.627897 17440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.628475 17440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.629647 17440 hybrid_clock.cc:648] HybridClock initialized: now 1790614113629587 us; error 71 us; skew 500 ppm
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.632529 17440 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:48:33.633395 17440 webserver.cc:460] Webserver started at http://127.30.196.2:38209/ using document root <none> and password file <none>
I20260928 16:48:33.633970 17440 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.634061 17440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.634290 17440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.636478 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "52c8f91c6ba34613b7d2d30210ebf2e3"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "f2219bbf5770369ae0e1213746cc4895"
server_key_iv: "1f6a9e0a2987569b455320dd9b039661"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.637069 17440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "52c8f91c6ba34613b7d2d30210ebf2e3"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "f2219bbf5770369ae0e1213746cc4895"
server_key_iv: "1f6a9e0a2987569b455320dd9b039661"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.641047 17440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:48:33.643733 17455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.645156 17440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:48:33.645319 17440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "52c8f91c6ba34613b7d2d30210ebf2e3"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "f2219bbf5770369ae0e1213746cc4895"
server_key_iv: "1f6a9e0a2987569b455320dd9b039661"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.645462 17440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.670167 17440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.673316 17440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.673478 17440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.674021 17440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:33.674799 17440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:33.674846 17440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.674887 17440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:33.674903 17440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.684386 17440 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:46063
I20260928 16:48:33.684404 17569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:46063 every 8 connection(s)
I20260928 16:48:33.685279 17440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:33.688221 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17440
I20260928 16:48:33.688337 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:33.688616 31504 external_mini_cluster.cc:1428] Setting key d80bb1957d5a1cb0cacb0b1d6ce662bf
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
I20260928 16:48:33.700461 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.687078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:38707 (local address 127.30.196.62:34561)
0928 16:48:33.687238 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.687243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.688354 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 16:48:33.689457 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.689460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.689524 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.689615 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.690741 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.691267 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.693127 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.693355 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.693436 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.696516 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.696519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.696521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.696549 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.698081 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.698619 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.698623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.698625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.698709 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.699124 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.699128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.699129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.699368 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.699488 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.700097 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.700236 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
I20260928 16:48:33.701364 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.687497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34561 (local address 127.30.196.2:38707)
0928 16:48:33.688196 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.688238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.689149 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.689784 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.689790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.690138 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:33.690604 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.690616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.691397 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.691403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.692946 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.692971 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.693516 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.693523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.693764 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.694540 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:33.694567 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:33.696309 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:33.698205 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.698211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.698230 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.698510 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.698810 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.698813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.698815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.698986 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.699488 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:33.699497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:33.699840 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.700038 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.700389 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":168,"threads_started":1}
I20260928 16:48:33.702668 17570 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34561
I20260928 16:48:33.702936 17570 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:33.703619 17570 heartbeater.cc:511] Master 127.30.196.62:34561 requested a full tablet report, sending...
I20260928 16:48:33.704833 17243 ts_manager.cc:194] Registered new tserver with Master: 52c8f91c6ba34613b7d2d30210ebf2e3 (127.30.196.2:46063)
I20260928 16:48:33.705489 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:38707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:48:33.719468 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.830013 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.830312 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.830381 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.834445 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.834537 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.834622 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:33.839931 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:34561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:33.841032 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.842077 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.852519 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.852550 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.852499 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.852731 17575 server_base.cc:1034] running on GCE node
I20260928 16:48:33.853287 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.854038 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.855248 17575 hybrid_clock.cc:648] HybridClock initialized: now 1790614113855225 us; error 27 us; skew 500 ppm
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.858199 17575 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:48:33.859196 17575 webserver.cc:460] Webserver started at http://127.30.196.3:37131/ using document root <none> and password file <none>
I20260928 16:48:33.859863 17575 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.859958 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.860195 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.862350 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "76223184ce3e420ebe3abfec24d97e8c"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "4898303ff92036837848d4c9c097f4a4"
server_key_iv: "b34eb799aaf2edae71b3fb0414914424"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.862939 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "76223184ce3e420ebe3abfec24d97e8c"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "4898303ff92036837848d4c9c097f4a4"
server_key_iv: "b34eb799aaf2edae71b3fb0414914424"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.866926 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 16:48:33.869745 17590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.871055 17575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:48:33.871196 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "76223184ce3e420ebe3abfec24d97e8c"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-5rg1"
server_key: "4898303ff92036837848d4c9c097f4a4"
server_key_iv: "b34eb799aaf2edae71b3fb0414914424"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.871327 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.884124 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.887567 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.887805 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.888451 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:33.889313 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:33.889391 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.889472 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:33.889520 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.898415 17575 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:39569
I20260928 16:48:33.898437 17704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:39569 every 8 connection(s)
I20260928 16:48:33.899363 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:33.906193 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17575
I20260928 16:48:33.906342 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:33.906684 31504 external_mini_cluster.cc:1428] Setting key 62b21a15d30a1ca95262fee3eabdde8e
Sep 28 16:48:33 dist-test-slave-5rg1 krb5kdc[17196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:33.912747 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.901273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:47915 (local address 127.30.196.62:34561)
0928 16:48:33.901415 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.901420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.902438 (+ 1018us) server_negotiation.cc:407] Connection header received
0928 16:48:33.903291 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.903293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.903342 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.903421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.904827 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.905379 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.906631 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.906938 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.907052 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.909571 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.909574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.909576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.909599 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.910784 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.911236 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.911239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.911240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.911315 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.911607 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.911610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.911611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.911834 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.911912 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.912431 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.912562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:33.913669 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.901644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:34561 (local address 127.30.196.3:47915)
0928 16:48:33.902307 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.902347 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.903105 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.903642 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.903648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.904179 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:33.904684 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.904698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.905505 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.905507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.906517 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.906528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.907122 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.907126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.907319 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.907958 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:33.907981 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:33.909420 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:33.910886 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.910891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.910904 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.911154 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.911390 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.911392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.911394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.911531 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.911921 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:33.911926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:33.912170 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.912334 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.912717 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":114,"threads_started":1}
I20260928 16:48:33.914786 17705 heartbeater.cc:347] Connected to a master server at 127.30.196.62:34561
I20260928 16:48:33.914995 17705 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:33.915436 17705 heartbeater.cc:511] Master 127.30.196.62:34561 requested a full tablet report, sending...
I20260928 16:48:33.916476 17243 ts_manager.cc:194] Registered new tserver with Master: 76223184ce3e420ebe3abfec24d97e8c (127.30.196.3:39569)
I20260928 16:48:33.917069 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:47915
I20260928 16:48:33.920367 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:33.930069 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.30.196.62:34561)
0928 16:48:33.922490 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.922495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.922703 (+ 208us) server_negotiation.cc:407] Connection header received
0928 16:48:33.922840 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.922843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.922886 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.922972 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.923720 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.924244 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.925077 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.925307 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.925411 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.926998 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.927002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.927005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.927039 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.928240 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.928735 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.928738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.928740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.928831 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.929158 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.929161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.929163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.929372 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.929528 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.929797 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.929896 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:48:33.932963 17243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:33.936003 17243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:33.950699 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.30.196.3:39569)
0928 16:48:33.945068 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.945081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.945577 (+ 496us) server_negotiation.cc:407] Connection header received
0928 16:48:33.945799 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.945805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.946017 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.946150 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.946673 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.947418 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.949036 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.949879 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.950008 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.950172 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.950407 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.950512 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":134,"threads_started":1}
I20260928 16:48:33.950702 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:39569 (local address 127.0.0.1:39652)
0928 16:48:33.945433 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.945494 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.945634 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.946338 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.946340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.946362 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.946510 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.946519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.947554 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.947560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.948916 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.948926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.950008 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.950012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.950138 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.950182 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.950354 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.950490 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":32,"spinlock_wait_cycles":13952,"thread_start_us":77,"threads_started":1}
I20260928 16:48:33.950950 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.30.196.1:40593)
0928 16:48:33.945335 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.945350 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.945760 (+ 410us) server_negotiation.cc:407] Connection header received
0928 16:48:33.945832 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.945836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.946020 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.946161 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.946482 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.947765 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.949312 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.950126 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.950233 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.950356 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.950670 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.950783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260928 16:48:33.951126 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40593 (local address 127.0.0.1:56798)
0928 16:48:33.945460 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.945493 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.945623 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.946180 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.946186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.946209 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.946358 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.946374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.947923 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.947926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.949184 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.949195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.950422 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.950426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.950523 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.950545 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.950623 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.950692 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260928 16:48:33.953708 17638 tablet_service.cc:1467] Processing CreateTablet for tablet 37175779e93941b1810a2c3d4147c3fe (DEFAULT_TABLE table=test-table [id=69321c95ba1f4bff9e0400a1bffd70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.953706 17369 tablet_service.cc:1467] Processing CreateTablet for tablet 37175779e93941b1810a2c3d4147c3fe (DEFAULT_TABLE table=test-table [id=69321c95ba1f4bff9e0400a1bffd70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.954666 17369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37175779e93941b1810a2c3d4147c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.954675 17638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37175779e93941b1810a2c3d4147c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.956238 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:46063 (local address 127.0.0.1:53646)
0928 16:48:33.945432 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.945484 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.945624 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.946332 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.946335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.946357 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.946478 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.946489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.948324 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.948327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.949401 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.949408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.955753 (+ 6345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.955757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.955863 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.955900 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.956007 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.956097 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":2,"spinlock_wait_cycles":2176,"thread_start_us":114,"threads_started":1}
I20260928 16:48:33.956696 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.944847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.30.196.2:46063)
0928 16:48:33.945322 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.945334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.945769 (+ 435us) server_negotiation.cc:407] Connection header received
0928 16:48:33.945858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.945865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.946060 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.946211 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.946605 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.947816 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.949508 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.950287 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.950390 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.950515 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.956426 (+ 5911us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.956546 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":118,"threads_started":1}
I20260928 16:48:33.959633 17504 tablet_service.cc:1467] Processing CreateTablet for tablet 37175779e93941b1810a2c3d4147c3fe (DEFAULT_TABLE table=test-table [id=69321c95ba1f4bff9e0400a1bffd70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.960875 17504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37175779e93941b1810a2c3d4147c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.961261 17723 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Bootstrap starting.
I20260928 16:48:33.961994 17724 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: Bootstrap starting.
I20260928 16:48:33.963471 17723 tablet_bootstrap.cc:654] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.964030 17724 tablet_bootstrap.cc:654] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.964640 17723 log.cc:824] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.965240 17724 log.cc:824] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.966626 17723 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: No bootstrap required, opened a new log
I20260928 16:48:33.966827 17723 ts_tablet_manager.cc:1397] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:48:33.967132 17724 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: No bootstrap required, opened a new log
I20260928 16:48:33.967306 17724 ts_tablet_manager.cc:1397] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:33.969112 17727 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Bootstrap starting.
I20260928 16:48:33.970295 17723 raft_consensus.cc:348] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.970579 17723 raft_consensus.cc:374] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.970633 17723 raft_consensus.cc:729] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76223184ce3e420ebe3abfec24d97e8c, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.970930 17724 raft_consensus.cc:348] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.971099 17723 consensus_queue.cc:260] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.971189 17724 raft_consensus.cc:374] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.971242 17724 raft_consensus.cc:729] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e42660b5ceb41de8f1f390b399d3c7e, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.971688 17724 consensus_queue.cc:260] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.971876 17705 heartbeater.cc:503] Master 127.30.196.62:34561 was elected leader, sending a full tablet report...
I20260928 16:48:33.972215 17723 ts_tablet_manager.cc:1428] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:33.972419 17727 tablet_bootstrap.cc:654] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.972606 17724 ts_tablet_manager.cc:1428] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260928 16:48:33.972869 17435 heartbeater.cc:503] Master 127.30.196.62:34561 was elected leader, sending a full tablet report...
I20260928 16:48:33.973433 17727 log.cc:824] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.975778 17727 tablet_bootstrap.cc:492] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: No bootstrap required, opened a new log
I20260928 16:48:33.975982 17727 ts_tablet_manager.cc:1397] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260928 16:48:33.979396 17727 raft_consensus.cc:348] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.979642 17727 raft_consensus.cc:374] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.979739 17727 raft_consensus.cc:729] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c8f91c6ba34613b7d2d30210ebf2e3, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.980165 17727 consensus_queue.cc:260] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:33.980808 17570 heartbeater.cc:503] Master 127.30.196.62:34561 was elected leader, sending a full tablet report...
I20260928 16:48:33.981112 17727 ts_tablet_manager.cc:1428] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 16:48:34.002631 17728 raft_consensus.cc:482] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:34.002883 17728 raft_consensus.cc:504] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:34.004245 17728 leader_election.cc:288] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c8f91c6ba34613b7d2d30210ebf2e3 (127.30.196.2:46063), 0e42660b5ceb41de8f1f390b399d3c7e (127.30.196.1:40593)
I20260928 16:48:34.008239 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.004502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40593 (local address 127.30.196.3:54207)
0928 16:48:34.004713 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.004729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.004834 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.005173 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.005176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.005196 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:34.005325 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.005335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.006175 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.006178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.007098 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.007108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.007796 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.007908 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.007938 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.008009 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.008072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 16:48:34.008266 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.004607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:54207 (local address 127.30.196.1:40593)
0928 16:48:34.004757 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.004764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.004781 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:48:34.004985 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.004988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.005052 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.005145 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:34.005444 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.006060 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007233 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.007639 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007845 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.007910 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.008041 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.008134 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:48:34.008348 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.004764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:55451 (local address 127.30.196.2:46063)
0928 16:48:34.004916 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.004923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.005144 (+ 221us) server_negotiation.cc:407] Connection header received
0928 16:48:34.005241 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.005244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.005298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.005386 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:34.005666 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.006219 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007268 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.007737 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007850 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.007907 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.008139 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.008213 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:48:34.008520 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.004514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:46063 (local address 127.30.196.3:55451)
0928 16:48:34.005044 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.005058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.005157 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.005420 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.005424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.005451 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:34.005554 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.005560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.006333 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.006335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.007128 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.007138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.007875 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.007878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.007985 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.008029 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.008107 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.008171 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":24,"thread_start_us":147,"threads_started":1}
I20260928 16:48:34.009074 17389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37175779e93941b1810a2c3d4147c3fe" candidate_uuid: "76223184ce3e420ebe3abfec24d97e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" is_pre_election: true
I20260928 16:48:34.009095 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37175779e93941b1810a2c3d4147c3fe" candidate_uuid: "76223184ce3e420ebe3abfec24d97e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" is_pre_election: true
I20260928 16:48:34.009404 17389 raft_consensus.cc:2454] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76223184ce3e420ebe3abfec24d97e8c in term 0.
I20260928 16:48:34.009404 17524 raft_consensus.cc:2454] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76223184ce3e420ebe3abfec24d97e8c in term 0.
I20260928 16:48:34.009941 17591 leader_election.cc:302] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c8f91c6ba34613b7d2d30210ebf2e3, 76223184ce3e420ebe3abfec24d97e8c; no voters:
I20260928 16:48:34.010260 17728 raft_consensus.cc:2781] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:34.010344 17728 raft_consensus.cc:482] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:34.010387 17728 raft_consensus.cc:3037] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.011459 17728 raft_consensus.cc:504] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:34.011845 17728 leader_election.cc:288] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [CANDIDATE]: Term 1 election: Requested vote from peers 52c8f91c6ba34613b7d2d30210ebf2e3 (127.30.196.2:46063), 0e42660b5ceb41de8f1f390b399d3c7e (127.30.196.1:40593)
I20260928 16:48:34.012198 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37175779e93941b1810a2c3d4147c3fe" candidate_uuid: "76223184ce3e420ebe3abfec24d97e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3"
I20260928 16:48:34.012290 17389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37175779e93941b1810a2c3d4147c3fe" candidate_uuid: "76223184ce3e420ebe3abfec24d97e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e42660b5ceb41de8f1f390b399d3c7e"
I20260928 16:48:34.012346 17524 raft_consensus.cc:3037] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.012431 17389 raft_consensus.cc:3037] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.013557 17389 raft_consensus.cc:2454] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76223184ce3e420ebe3abfec24d97e8c in term 1.
I20260928 16:48:34.013597 17524 raft_consensus.cc:2454] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76223184ce3e420ebe3abfec24d97e8c in term 1.
I20260928 16:48:34.013913 17591 leader_election.cc:302] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c8f91c6ba34613b7d2d30210ebf2e3, 76223184ce3e420ebe3abfec24d97e8c; no voters:
I20260928 16:48:34.014139 17728 raft_consensus.cc:2781] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:34.014379 17728 raft_consensus.cc:686] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 1 LEADER]: Becoming Leader. State: Replica: 76223184ce3e420ebe3abfec24d97e8c, State: Running, Role: LEADER
I20260928 16:48:34.014767 17728 consensus_queue.cc:237] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } }
I20260928 16:48:34.018083 17242 catalog_manager.cc:5495] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 76223184ce3e420ebe3abfec24d97e8c (127.30.196.3). New cstate: current_term: 1 leader_uuid: "76223184ce3e420ebe3abfec24d97e8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76223184ce3e420ebe3abfec24d97e8c" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 39569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 } health_report { overall_health: UNKNOWN } } }
I20260928 16:48:34.034044 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.030522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.30.196.3:39569)
0928 16:48:34.030641 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.030644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.030658 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:48:34.030782 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.030784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.030838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.030925 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:34.031181 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.031727 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.032595 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.032797 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.032872 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.033190 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:34.033619 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:34.033728 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.033861 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.033897 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:48:34.039951 17524 raft_consensus.cc:1261] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 1 FOLLOWER]: Refusing update from remote peer 76223184ce3e420ebe3abfec24d97e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:34.039968 17389 raft_consensus.cc:1261] T 37175779e93941b1810a2c3d4147c3fe P 0e42660b5ceb41de8f1f390b399d3c7e [term 1 FOLLOWER]: Refusing update from remote peer 76223184ce3e420ebe3abfec24d97e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:34.040679 17728 consensus_queue.cc:1035] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e42660b5ceb41de8f1f390b399d3c7e" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:34.040910 17733 consensus_queue.cc:1035] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c8f91c6ba34613b7d2d30210ebf2e3" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:34.048884 17737 mvcc.cc:205] Tried to move back new op lower bound from 7334355411100430336 to 7334355411007299584. Current Snapshot: MvccSnapshot[applied={T|T < 7334355411100430336}]
I20260928 16:48:34.049247 17736 mvcc.cc:205] Tried to move back new op lower bound from 7334355411100430336 to 7334355411007299584. Current Snapshot: MvccSnapshot[applied={T|T < 7334355411100430336}]
I20260928 16:48:34.059765 17243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:34.059955 17243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:34.062340 17243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8cf4f498aa44c5c8e34029f15aff254: Sending DeleteTablet for 3 replicas of tablet 37175779e93941b1810a2c3d4147c3fe
I20260928 16:48:34.063087 17638 tablet_service.cc:1514] Processing DeleteTablet for tablet 37175779e93941b1810a2c3d4147c3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:39652
I20260928 16:48:34.063313 17504 tablet_service.cc:1514] Processing DeleteTablet for tablet 37175779e93941b1810a2c3d4147c3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:53646
I20260928 16:48:34.063472 17369 tablet_service.cc:1514] Processing DeleteTablet for tablet 37175779e93941b1810a2c3d4147c3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:56798
I20260928 16:48:34.064030 17746 tablet_replica.cc:330] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: stopping tablet replica
I20260928 16:48:34.064399 17746 raft_consensus.cc:2229] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:34.064540 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17305
I20260928 16:48:34.064822 17746 raft_consensus.cc:2258] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:34.066752 17746 ts_tablet_manager.cc:1905] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:34.068120 17747 tablet_replica.cc:330] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: stopping tablet replica
I20260928 16:48:34.068495 17747 raft_consensus.cc:2229] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:34.068869 17747 raft_consensus.cc:2258] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:34.070183 17746 ts_tablet_manager.cc:1918] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:34.070338 17746 log.cc:1196] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/37175779e93941b1810a2c3d4147c3fe
I20260928 16:48:34.070498 17747 ts_tablet_manager.cc:1905] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:34.070863 17746 ts_tablet_manager.cc:1939] T 37175779e93941b1810a2c3d4147c3fe P 76223184ce3e420ebe3abfec24d97e8c: Deleting consensus metadata
I20260928 16:48:34.072242 17228 catalog_manager.cc:4841] TS 76223184ce3e420ebe3abfec24d97e8c (127.30.196.3:39569): tablet 37175779e93941b1810a2c3d4147c3fe (table test-table [id=69321c95ba1f4bff9e0400a1bffd70fa]) successfully deleted
I20260928 16:48:34.073273 17747 ts_tablet_manager.cc:1918] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:34.073400 17747 log.cc:1196] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/37175779e93941b1810a2c3d4147c3fe
I20260928 16:48:34.073699 17747 ts_tablet_manager.cc:1939] T 37175779e93941b1810a2c3d4147c3fe P 52c8f91c6ba34613b7d2d30210ebf2e3: Deleting consensus metadata
W20260928 16:48:34.074153 17230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:40593 (error 108)
I20260928 16:48:34.075054 17227 catalog_manager.cc:4841] TS 52c8f91c6ba34613b7d2d30210ebf2e3 (127.30.196.2:46063): tablet 37175779e93941b1810a2c3d4147c3fe (table test-table [id=69321c95ba1f4bff9e0400a1bffd70fa]) successfully deleted
I20260928 16:48:34.075232 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.074899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40593 (local address 127.0.0.1:56810)
0928 16:48:34.075081 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.075144 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 16:48:34.075376 17230 catalog_manager.cc:4584] TS 0e42660b5ceb41de8f1f390b399d3c7e (127.30.196.1:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37175779e93941b1810a2c3d4147c3fe: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40593: connect: Connection refused (error 111)
I20260928 16:48:34.075845 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17440
I20260928 16:48:34.082370 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17575
I20260928 16:48:34.088593 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17210
2026-09-28T16:48:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2328 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T16:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:34Z Disabled control of system clock
I20260928 16:48:34.120169 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33135
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:34.244673 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:34.244956 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:34.244994 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:34.249275 17753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:34.249343 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:34.249364 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:34.249382 17753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:34.249435 17753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:34.254299 17753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:33135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:34.255491 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:34.257014 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:34.265906 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.265882 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.265983 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:34.266348 17753 server_base.cc:1034] running on GCE node
I20260928 16:48:34.266916 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:34.268151 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:34.269344 17753 hybrid_clock.cc:648] HybridClock initialized: now 1790614114269326 us; error 32 us; skew 500 ppm
I20260928 16:48:34.271298 17753 webserver.cc:460] Webserver started at http://127.30.196.62:40285/ using document root <none> and password file <none>
I20260928 16:48:34.271975 17753 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:34.272078 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:34.272320 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:34.274421 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "2bbd6710a050496f94a954b5f37d5a68"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "655e82872161a1a6d9c4eb186040e62f"
server_key_iv: "8e0b15451af87233353eadb7b91d88be"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.275008 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "2bbd6710a050496f94a954b5f37d5a68"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "655e82872161a1a6d9c4eb186040e62f"
server_key_iv: "8e0b15451af87233353eadb7b91d88be"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.278782 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:34.281383 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.282770 17753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:34.283022 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "2bbd6710a050496f94a954b5f37d5a68"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "655e82872161a1a6d9c4eb186040e62f"
server_key_iv: "8e0b15451af87233353eadb7b91d88be"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.283150 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:34.311810 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:34.312633 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:34.312849 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:34.320333 17753 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:33135
I20260928 16:48:34.320379 17821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:33135 every 8 connection(s)
I20260928 16:48:34.321339 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:34.324826 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:34.327241 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17753
I20260928 16:48:34.327349 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:34.327654 31504 external_mini_cluster.cc:1428] Setting key 4f74a8ad0b4b8b8cf3eec1324a6acc05
I20260928 16:48:34.331853 17822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: Bootstrap starting.
I20260928 16:48:34.334615 17822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:34.335398 17822 log.cc:824] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:34.337563 17822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: No bootstrap required, opened a new log
I20260928 16:48:34.338199 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.329218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.30.196.62:33135)
0928 16:48:34.329918 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.329938 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.329986 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:48:34.330871 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.330896 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.331417 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.331818 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.332293 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.333916 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.334788 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.335261 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.335446 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.336347 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.336354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.336367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:34.336791 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.336853 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:34.336874 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.336957 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.337131 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.337432 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260928 16:48:34.341169 17822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } }
I20260928 16:48:34.341390 17822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:34.341449 17822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbd6710a050496f94a954b5f37d5a68, State: Initialized, Role: FOLLOWER
I20260928 16:48:34.341878 17822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } }
I20260928 16:48:34.342005 17822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:34.342090 17822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:34.342216 17822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.343150 17822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } }
I20260928 16:48:34.343494 17822 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bbd6710a050496f94a954b5f37d5a68; no voters:
I20260928 16:48:34.343848 17822 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:34.344046 17827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:34.344308 17827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [term 1 LEADER]: Becoming Leader. State: Replica: 2bbd6710a050496f94a954b5f37d5a68, State: Running, Role: LEADER
I20260928 16:48:34.344615 17827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } }
I20260928 16:48:34.345088 17822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:34.346282 17829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbd6710a050496f94a954b5f37d5a68. Latest consensus state: current_term: 1 leader_uuid: "2bbd6710a050496f94a954b5f37d5a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } } }
I20260928 16:48:34.346406 17829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:34.346746 17828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bbd6710a050496f94a954b5f37d5a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd6710a050496f94a954b5f37d5a68" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33135 } } }
I20260928 16:48:34.346851 17828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:34.347791 17836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:34.350541 17836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:34.356462 17836 catalog_manager.cc:1347] Generated new cluster ID: 7db0712aceec4d9eb7314868825e5e52
I20260928 16:48:34.356556 17836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:34.371973 17836 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:34.373263 17836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:34.385002 17836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: Generated new TSK 0
I20260928 16:48:34.385757 17836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:34.395846 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--builtin_ntp_servers=127.30.196.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:34.535463 17846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:34.535808 17846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:34.535882 17846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:34.540046 17846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:34.540119 17846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:34.540210 17846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:34.545171 17846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:34.546506 17846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:34.548141 17846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:34.558099 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.558221 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.558123 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:34.558614 17846 server_base.cc:1034] running on GCE node
I20260928 16:48:34.559249 17846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:34.560259 17846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:34.561440 17846 hybrid_clock.cc:648] HybridClock initialized: now 1790614114561409 us; error 47 us; skew 500 ppm
I20260928 16:48:34.564103 17846 webserver.cc:460] Webserver started at http://127.30.196.1:40741/ using document root <none> and password file <none>
I20260928 16:48:34.564906 17846 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:34.564991 17846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:34.565272 17846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:34.567622 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "e8a19f684f7e46e79b355560f2c42a30"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "6c19fbe08fb8959f9c55c0d2ad7304ef"
server_key_iv: "d72d0a0b103dcda0d6f49079cf2b871f"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.568228 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "e8a19f684f7e46e79b355560f2c42a30"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "6c19fbe08fb8959f9c55c0d2ad7304ef"
server_key_iv: "d72d0a0b103dcda0d6f49079cf2b871f"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.572119 17846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:34.574479 17861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.575819 17846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:34.575950 17846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "e8a19f684f7e46e79b355560f2c42a30"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "6c19fbe08fb8959f9c55c0d2ad7304ef"
server_key_iv: "d72d0a0b103dcda0d6f49079cf2b871f"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.576056 17846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:34.593593 17846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:34.594411 17846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:34.594640 17846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:34.595289 17846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:34.596415 17846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:34.596464 17846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.596505 17846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:34.596520 17846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.605583 17846 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:37443
I20260928 16:48:34.605612 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:37443 every 8 connection(s)
I20260928 16:48:34.606580 17846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:34.615943 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17846
I20260928 16:48:34.616111 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:34.616158 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.608539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:37253 (local address 127.30.196.62:33135)
0928 16:48:34.608789 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.608793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.609738 (+ 945us) server_negotiation.cc:407] Connection header received
0928 16:48:34.611188 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.611191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.611263 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.611372 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.612349 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.612907 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.614119 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.614350 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.614504 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.615609 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.615614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.615623 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:34.615637 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.615657 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:34.615711 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.615778 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.615941 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.616011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 16:48:34.616470 31504 external_mini_cluster.cc:1428] Setting key 4633d1caa592bfb5b67feaf887592ec5
I20260928 16:48:34.617136 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.608913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33135 (local address 127.30.196.1:37253)
0928 16:48:34.609585 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.609625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.610961 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.611558 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.611564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.611648 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:34.612208 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.612221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.613048 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.613054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.613986 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.613996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.614553 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.614557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.614731 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.615337 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:34.615338 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:34.615379 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:34.615453 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:34.615780 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:34.615793 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:34.615799 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.615938 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.616230 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260928 16:48:34.618517 17975 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33135
I20260928 16:48:34.618726 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--builtin_ntp_servers=127.30.196.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:34.618851 17975 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:34.619462 17975 heartbeater.cc:511] Master 127.30.196.62:33135 requested a full tablet report, sending...
I20260928 16:48:34.621241 17786 ts_manager.cc:194] Registered new tserver with Master: e8a19f684f7e46e79b355560f2c42a30 (127.30.196.1:37443)
I20260928 16:48:34.622421 17786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.1:37253
W20260928 16:48:34.732573 17978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:34.732899 17978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:34.732939 17978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:34.737252 17978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:34.737310 17978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:34.737391 17978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:34.742372 17978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:34.743568 17978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:34.745057 17978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:34.755259 17984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.755312 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.755277 17983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:34.755641 17978 server_base.cc:1034] running on GCE node
I20260928 16:48:34.756126 17978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:34.756780 17978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:34.757979 17978 hybrid_clock.cc:648] HybridClock initialized: now 1790614114757969 us; error 33 us; skew 500 ppm
I20260928 16:48:34.759931 17978 webserver.cc:460] Webserver started at http://127.30.196.2:38891/ using document root <none> and password file <none>
I20260928 16:48:34.760528 17978 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:34.760620 17978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:34.760882 17978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:34.762950 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "a587581754694eff97e7989e1f2b0d1d"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "075bb42d65b3003ae3b1cdafcfc2a1ca"
server_key_iv: "831babee63f6644871889fb7c5a34a8b"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.763530 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "a587581754694eff97e7989e1f2b0d1d"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "075bb42d65b3003ae3b1cdafcfc2a1ca"
server_key_iv: "831babee63f6644871889fb7c5a34a8b"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.767311 17978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:48:34.769968 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.771230 17978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:48:34.771391 17978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "a587581754694eff97e7989e1f2b0d1d"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "075bb42d65b3003ae3b1cdafcfc2a1ca"
server_key_iv: "831babee63f6644871889fb7c5a34a8b"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.771519 17978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:34.786219 17978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:34.787016 17978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:34.787222 17978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:34.787949 17978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:34.788982 17978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:34.789031 17978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.789099 17978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:34.789136 17978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.797889 17978 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:42555
I20260928 16:48:34.797952 18106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:42555 every 8 connection(s)
I20260928 16:48:34.798800 17978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:34.807626 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 17978
I20260928 16:48:34.807796 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:34.808109 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.800616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:37803 (local address 127.30.196.62:33135)
0928 16:48:34.800755 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.800760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.801780 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 16:48:34.803203 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.803206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.803253 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.803341 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.804319 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.804848 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.805967 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.806211 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.806320 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.807436 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.807439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.807441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:34.807455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.807474 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:34.807483 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.807551 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.807895 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.807975 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:48:34.808125 31504 external_mini_cluster.cc:1428] Setting key 2d719e074f992a10c99be785e5e88be0
I20260928 16:48:34.809115 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.800999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33135 (local address 127.30.196.2:37803)
0928 16:48:34.801633 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.801672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.802917 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.803495 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.803502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.803583 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:34.804164 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.804178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.804955 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.804958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.805826 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.805835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.806365 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.806367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.806504 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.807143 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:34.807145 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:34.807184 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:34.807246 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:34.807581 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:34.807591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:34.807597 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.807838 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.808171 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260928 16:48:34.810319 18107 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33135
I20260928 16:48:34.810570 18107 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:34.810668 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--builtin_ntp_servers=127.30.196.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:34.811196 18107 heartbeater.cc:511] Master 127.30.196.62:33135 requested a full tablet report, sending...
I20260928 16:48:34.812400 17781 ts_manager.cc:194] Registered new tserver with Master: a587581754694eff97e7989e1f2b0d1d (127.30.196.2:42555)
I20260928 16:48:34.812990 17781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.2:37803
W20260928 16:48:34.924791 18110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:34.925137 18110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:34.925210 18110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:34.929293 18110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:34.929378 18110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:34.929469 18110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:34.934221 18110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:34.935343 18110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:34.936731 18110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:34.946282 18118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:34.946282 18115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:34.946628 18110 server_base.cc:1034] running on GCE node
W20260928 16:48:34.947177 18116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:34.947621 18110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:34.948274 18110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:34.949470 18110 hybrid_clock.cc:648] HybridClock initialized: now 1790614114949458 us; error 36 us; skew 500 ppm
I20260928 16:48:34.951472 18110 webserver.cc:460] Webserver started at http://127.30.196.3:44515/ using document root <none> and password file <none>
I20260928 16:48:34.952165 18110 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:34.952266 18110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:34.952489 18110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:34.954779 18110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "6baf9937707e48579e9ea086578257d7"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "fdebea62b3df98c5402136ce4ed006c2"
server_key_iv: "7360882e176491861c9a0c363652c990"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.955371 18110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "6baf9937707e48579e9ea086578257d7"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "fdebea62b3df98c5402136ce4ed006c2"
server_key_iv: "7360882e176491861c9a0c363652c990"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.959256 18110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 16:48:34.961720 18124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.962929 18110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:34.963065 18110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "6baf9937707e48579e9ea086578257d7"
format_stamp: "Formatted at 2026-09-28 16:48:34 on dist-test-slave-5rg1"
server_key: "fdebea62b3df98c5402136ce4ed006c2"
server_key_iv: "7360882e176491861c9a0c363652c990"
server_key_version: "encryptionkey@0"
I20260928 16:48:34.963179 18110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:34.987115 18110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:34.988045 18110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:34.988260 18110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:34.988922 18110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:34.989916 18110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:34.989997 18110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.990070 18110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:34.990120 18110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:34.998871 18110 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:41913
I20260928 16:48:34.998899 18238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:41913 every 8 connection(s)
I20260928 16:48:34.999859 18110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:35.008144 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18110
I20260928 16:48:35.008294 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:35.008692 31504 external_mini_cluster.cc:1428] Setting key d7c1c04899f5b2ef6a0b1ce464fa2ce8
I20260928 16:48:35.009466 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.001724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:58585 (local address 127.30.196.62:33135)
0928 16:48:35.001878 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.001882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.002938 (+ 1056us) server_negotiation.cc:407] Connection header received
0928 16:48:35.004414 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.004417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.004488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.004598 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:35.005509 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.006044 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.007198 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.007431 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.007554 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.008941 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:35.008944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:35.008946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:35.008967 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:35.008989 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:35.008997 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:35.009091 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.009254 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.009326 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:48:35.010475 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.002133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33135 (local address 127.30.196.3:58585)
0928 16:48:35.002781 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.002821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.004194 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.004782 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.004788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.004872 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:35.005359 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.005370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.006187 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.006190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.007068 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.007077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.007601 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.007605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.007843 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.008591 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:35.008594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:35.008653 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:35.008744 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:35.009104 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:35.009112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:35.009117 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.009224 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.009585 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260928 16:48:35.011552 18239 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33135
I20260928 16:48:35.011823 18239 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:35.012362 18239 heartbeater.cc:511] Master 127.30.196.62:33135 requested a full tablet report, sending...
I20260928 16:48:35.013345 17781 ts_manager.cc:194] Registered new tserver with Master: 6baf9937707e48579e9ea086578257d7 (127.30.196.3:41913)
I20260928 16:48:35.013938 17781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.3:58585
I20260928 16:48:35.023031 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:35.029265 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.025117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.30.196.62:33135)
0928 16:48:35.025281 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.025286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.025435 (+ 149us) server_negotiation.cc:407] Connection header received
0928 16:48:35.025613 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.025616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.025657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.025741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:35.026072 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.026597 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.027464 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.027727 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.027826 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.028651 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:35.028655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:35.028658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:35.028671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:35.028696 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:35.028703 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:35.028846 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.029072 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.029116 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:48:35.032167 17781 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:35.035279 17781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:35.050621 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.044350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:37443 (local address 127.0.0.1:41818)
0928 16:48:35.045077 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.045135 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.045274 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.046181 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.046188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.046211 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:35.046368 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.046376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.047753 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.047756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.049024 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.049033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.050036 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.050148 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050195 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.050297 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.050415 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"spinlock_wait_cycles":7680,"thread_start_us":122,"threads_started":1}
I20260928 16:48:35.050815 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.044674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41913 (local address 127.0.0.1:54734)
0928 16:48:35.045519 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.045531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.045607 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.046547 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.046553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.046583 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:35.046661 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.046667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.048061 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.048064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.049415 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.049426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.050336 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.050433 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050467 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.050539 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.050615 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":102,"threads_started":1}
I20260928 16:48:35.051023 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.044204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:42555 (local address 127.0.0.1:42448)
0928 16:48:35.045077 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.045135 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.045274 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.046364 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.046367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.046391 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:35.046506 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.046513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.048052 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.048056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.049376 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.049388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.050626 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.050714 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050736 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.050803 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.050875 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":93,"spinlock_wait_cycles":2816,"thread_start_us":95,"threads_started":1}
I20260928 16:48:35.051083 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.045536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.30.196.3:41913)
0928 16:48:35.046042 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.046054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.046080 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:48:35.046156 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.046161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.046329 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.046467 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:35.046778 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.047938 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.049552 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.050205 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050472 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.050563 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.050666 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":195,"threads_started":1}
I20260928 16:48:35.051508 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.044344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.30.196.2:42555)
0928 16:48:35.045868 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.045880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.045903 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:48:35.045989 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.045992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.046219 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.046367 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:35.046641 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.047915 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.049515 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.050297 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050569 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.051236 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.051345 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":133,"threads_started":1}
I20260928 16:48:35.051537 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.045112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.30.196.1:37443)
0928 16:48:35.045620 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.045631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.045652 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:35.045717 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.045720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.046002 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.046163 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:35.046503 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.047582 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.049151 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.049864 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.050007 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.050202 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.051189 (+ 987us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.051370 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":148,"threads_started":1}
I20260928 16:48:35.054175 18041 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 (DEFAULT_TABLE table=test-table [id=b07ab408891b433bb440fab38db72a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:35.054311 17909 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 (DEFAULT_TABLE table=test-table [id=b07ab408891b433bb440fab38db72a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:35.054473 18173 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 (DEFAULT_TABLE table=test-table [id=b07ab408891b433bb440fab38db72a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:35.055197 18041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a59bf6ba447b992f4f9790c8882f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:35.055323 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a59bf6ba447b992f4f9790c8882f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:35.055337 17909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a59bf6ba447b992f4f9790c8882f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:35.061892 18256 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Bootstrap starting.
I20260928 16:48:35.063172 18258 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Bootstrap starting.
I20260928 16:48:35.064018 18257 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Bootstrap starting.
I20260928 16:48:35.064534 18256 tablet_bootstrap.cc:654] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:35.065196 18258 tablet_bootstrap.cc:654] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:35.065299 18256 log.cc:824] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:35.065949 18257 tablet_bootstrap.cc:654] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:35.065963 18258 log.cc:824] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:35.066741 18257 log.cc:824] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:35.067497 18256 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: No bootstrap required, opened a new log
I20260928 16:48:35.067742 18256 ts_tablet_manager.cc:1397] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:48:35.068025 18258 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: No bootstrap required, opened a new log
I20260928 16:48:35.068218 18258 ts_tablet_manager.cc:1397] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:35.068579 18257 tablet_bootstrap.cc:492] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: No bootstrap required, opened a new log
I20260928 16:48:35.068763 18257 ts_tablet_manager.cc:1397] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:35.071798 18257 raft_consensus.cc:348] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.072088 18257 raft_consensus.cc:374] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:35.072140 18257 raft_consensus.cc:729] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baf9937707e48579e9ea086578257d7, State: Initialized, Role: FOLLOWER
I20260928 16:48:35.072170 18256 raft_consensus.cc:348] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.072252 18258 raft_consensus.cc:348] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.072407 18256 raft_consensus.cc:374] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:35.072454 18256 raft_consensus.cc:729] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a587581754694eff97e7989e1f2b0d1d, State: Initialized, Role: FOLLOWER
I20260928 16:48:35.072443 18258 raft_consensus.cc:374] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:35.072522 18258 raft_consensus.cc:729] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a19f684f7e46e79b355560f2c42a30, State: Initialized, Role: FOLLOWER
I20260928 16:48:35.072638 18257 consensus_queue.cc:260] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.072885 18258 consensus_queue.cc:260] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.072886 18256 consensus_queue.cc:260] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.073462 18239 heartbeater.cc:503] Master 127.30.196.62:33135 was elected leader, sending a full tablet report...
I20260928 16:48:35.073691 18257 ts_tablet_manager.cc:1428] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:35.073797 18258 ts_tablet_manager.cc:1428] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:48:35.074142 17975 heartbeater.cc:503] Master 127.30.196.62:33135 was elected leader, sending a full tablet report...
I20260928 16:48:35.074877 18256 ts_tablet_manager.cc:1428] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:48:35.075201 18107 heartbeater.cc:503] Master 127.30.196.62:33135 was elected leader, sending a full tablet report...
W20260928 16:48:35.107841 17976 tablet.cc:2369] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:35.199165 18262 raft_consensus.cc:482] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:35.199380 18262 raft_consensus.cc:504] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.200698 18262 leader_election.cc:288] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a587581754694eff97e7989e1f2b0d1d (127.30.196.2:42555), e8a19f684f7e46e79b355560f2c42a30 (127.30.196.1:37443)
I20260928 16:48:35.206315 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.201103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:47497 (local address 127.30.196.2:42555)
0928 16:48:35.201235 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.201240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.201569 (+ 329us) server_negotiation.cc:407] Connection header received
0928 16:48:35.201739 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.201742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.201799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.201850 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:35.202288 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.203110 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.204920 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.205531 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.205657 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.205708 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.206107 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.206179 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:48:35.206315 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.201235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:45513 (local address 127.30.196.1:37443)
0928 16:48:35.201382 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.201387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.201627 (+ 240us) server_negotiation.cc:407] Connection header received
0928 16:48:35.201699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.201705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.201762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.201833 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:35.202288 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.203083 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.204920 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.205531 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.205657 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.205708 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.206107 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.206180 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:48:35.206338 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.201042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:37443 (local address 127.30.196.3:45513)
0928 16:48:35.201474 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.201491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.201588 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.201958 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.201961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.201980 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:35.202156 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.202163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.203229 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.203233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.204780 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.204792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.205677 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.205681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.205858 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.205992 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.206080 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.206175 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":300,"spinlock_wait_cycles":4992,"thread_start_us":140,"threads_started":1}
I20260928 16:48:35.206346 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.200942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:42555 (local address 127.30.196.3:47497)
0928 16:48:35.201465 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.201477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.201608 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.201986 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.201989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.202016 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:35.202169 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.202174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.203229 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.203233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.204787 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.204792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.205677 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.205681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.205839 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.205992 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.206080 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.206175 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":206}
I20260928 16:48:35.207105 17929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a59bf6ba447b992f4f9790c8882f2" candidate_uuid: "6baf9937707e48579e9ea086578257d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a19f684f7e46e79b355560f2c42a30" is_pre_election: true
I20260928 16:48:35.207116 18061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a59bf6ba447b992f4f9790c8882f2" candidate_uuid: "6baf9937707e48579e9ea086578257d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a587581754694eff97e7989e1f2b0d1d" is_pre_election: true
I20260928 16:48:35.207436 18061 raft_consensus.cc:2454] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baf9937707e48579e9ea086578257d7 in term 0.
I20260928 16:48:35.207438 17929 raft_consensus.cc:2454] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baf9937707e48579e9ea086578257d7 in term 0.
I20260928 16:48:35.207980 18125 leader_election.cc:302] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6baf9937707e48579e9ea086578257d7, a587581754694eff97e7989e1f2b0d1d, e8a19f684f7e46e79b355560f2c42a30; no voters:
I20260928 16:48:35.208268 18262 raft_consensus.cc:2781] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:35.208369 18262 raft_consensus.cc:482] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:35.208431 18262 raft_consensus.cc:3037] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:35.209474 18262 raft_consensus.cc:504] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.209859 18262 leader_election.cc:288] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [CANDIDATE]: Term 1 election: Requested vote from peers a587581754694eff97e7989e1f2b0d1d (127.30.196.2:42555), e8a19f684f7e46e79b355560f2c42a30 (127.30.196.1:37443)
I20260928 16:48:35.210233 17929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a59bf6ba447b992f4f9790c8882f2" candidate_uuid: "6baf9937707e48579e9ea086578257d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a19f684f7e46e79b355560f2c42a30"
I20260928 16:48:35.210207 18061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a59bf6ba447b992f4f9790c8882f2" candidate_uuid: "6baf9937707e48579e9ea086578257d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a587581754694eff97e7989e1f2b0d1d"
I20260928 16:48:35.210362 17929 raft_consensus.cc:3037] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:35.210366 18061 raft_consensus.cc:3037] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:35.211357 17929 raft_consensus.cc:2454] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baf9937707e48579e9ea086578257d7 in term 1.
I20260928 16:48:35.211411 18061 raft_consensus.cc:2454] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baf9937707e48579e9ea086578257d7 in term 1.
I20260928 16:48:35.211759 18127 leader_election.cc:302] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6baf9937707e48579e9ea086578257d7, e8a19f684f7e46e79b355560f2c42a30; no voters:
I20260928 16:48:35.211941 18262 raft_consensus.cc:2781] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:35.212179 18262 raft_consensus.cc:686] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6baf9937707e48579e9ea086578257d7, State: Running, Role: LEADER
I20260928 16:48:35.212507 18262 consensus_queue.cc:237] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } }
I20260928 16:48:35.215652 17781 catalog_manager.cc:5495] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6baf9937707e48579e9ea086578257d7 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "6baf9937707e48579e9ea086578257d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6baf9937707e48579e9ea086578257d7" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41913 } health_report { overall_health: HEALTHY } } }
I20260928 16:48:35.247892 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.243910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.30.196.3:41913)
0928 16:48:35.244092 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.244096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.244107 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:48:35.244178 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.244180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.244236 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.244348 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:35.244605 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.245107 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.246106 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.246324 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.246474 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.246815 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:35.247298 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:35.247411 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.247624 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.247677 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 16:48:35.250699 18240 tablet.cc:2369] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:35.254179 17929 raft_consensus.cc:1261] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 1 FOLLOWER]: Refusing update from remote peer 6baf9937707e48579e9ea086578257d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:35.254208 18061 raft_consensus.cc:1261] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 1 FOLLOWER]: Refusing update from remote peer 6baf9937707e48579e9ea086578257d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:35.254853 18262 consensus_queue.cc:1035] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a19f684f7e46e79b355560f2c42a30" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:35.255110 18266 consensus_queue.cc:1035] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a587581754694eff97e7989e1f2b0d1d" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:35.273342 17786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:35.273548 17786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:35.276412 17786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bbd6710a050496f94a954b5f37d5a68: Sending DeleteTablet for 3 replicas of tablet ab7a59bf6ba447b992f4f9790c8882f2
I20260928 16:48:35.277263 18041 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:35 UTC) from {username='slave'} at 127.0.0.1:42448
I20260928 16:48:35.277325 17909 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:35 UTC) from {username='slave'} at 127.0.0.1:41818
I20260928 16:48:35.277767 18281 tablet_replica.cc:330] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: stopping tablet replica
I20260928 16:48:35.277891 18283 tablet_replica.cc:330] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30: stopping tablet replica
I20260928 16:48:35.278263 18283 raft_consensus.cc:2229] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:35.278262 18281 raft_consensus.cc:2229] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:35.278618 18283 raft_consensus.cc:2258] T ab7a59bf6ba447b992f4f9790c8882f2 P e8a19f684f7e46e79b355560f2c42a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:35.278636 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17846
I20260928 16:48:35.278764 18281 raft_consensus.cc:2258] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:35.281514 18281 ts_tablet_manager.cc:1905] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:35.282352 18173 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7a59bf6ba447b992f4f9790c8882f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:35 UTC) from {username='slave'} at 127.0.0.1:54734
I20260928 16:48:35.283876 18284 tablet_replica.cc:330] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: stopping tablet replica
I20260928 16:48:35.284189 18284 raft_consensus.cc:2229] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:35.284600 18284 raft_consensus.cc:2258] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:35.286113 18284 ts_tablet_manager.cc:1905] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:35.286139 18281 ts_tablet_manager.cc:1918] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:35.286362 18281 log.cc:1196] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/ab7a59bf6ba447b992f4f9790c8882f2
I20260928 16:48:35.286872 18281 ts_tablet_manager.cc:1939] T ab7a59bf6ba447b992f4f9790c8882f2 P a587581754694eff97e7989e1f2b0d1d: Deleting consensus metadata
I20260928 16:48:35.288609 17769 catalog_manager.cc:4841] TS a587581754694eff97e7989e1f2b0d1d (127.30.196.2:42555): tablet ab7a59bf6ba447b992f4f9790c8882f2 (table test-table [id=b07ab408891b433bb440fab38db72a96]) successfully deleted
I20260928 16:48:35.288847 18284 ts_tablet_manager.cc:1918] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 16:48:35.288867 17771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:37443 (error 108)
I20260928 16:48:35.288931 18284 log.cc:1196] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/ab7a59bf6ba447b992f4f9790c8882f2
I20260928 16:48:35.289233 18284 ts_tablet_manager.cc:1939] T ab7a59bf6ba447b992f4f9790c8882f2 P 6baf9937707e48579e9ea086578257d7: Deleting consensus metadata
I20260928 16:48:35.289794 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17978
I20260928 16:48:35.290683 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.290220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:37443 (local address 127.0.0.1:41834)
0928 16:48:35.290476 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.290563 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260928 16:48:35.291062 17771 catalog_manager.cc:4584] TS e8a19f684f7e46e79b355560f2c42a30 (127.30.196.1:37443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab7a59bf6ba447b992f4f9790c8882f2: Network error: Client connection negotiation failed: client connection to 127.30.196.1:37443: connect: Connection refused (error 111)
I20260928 16:48:35.292588 17772 catalog_manager.cc:4841] TS 6baf9937707e48579e9ea086578257d7 (127.30.196.3:41913): tablet ab7a59bf6ba447b992f4f9790c8882f2 (table test-table [id=b07ab408891b433bb440fab38db72a96]) successfully deleted
I20260928 16:48:35.296223 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18110
I20260928 16:48:35.302961 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 17753
2026-09-28T16:48:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:35 dist-test-slave-5rg1 krb5kdc[18288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:35 dist-test-slave-5rg1 krb5kdc[18288](info): set up 2 sockets
Sep 28 16:48:35 dist-test-slave-5rg1 krb5kdc[18288](info): commencing operation
krb5kdc: starting...
W20260928 16:48:36.340164 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:36 dist-test-slave-5rg1 krb5kdc[18288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:48:36.467252 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33141
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:36.581928 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:36.582229 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:36.582271 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:36.586346 18301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:36.586413 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:36.586432 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:36.586449 18301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:36.586465 18301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:36.592027 18301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:36.593139 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:36.594172 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:36.603183 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:36.603338 18301 server_base.cc:1034] running on GCE node
W20260928 16:48:36.603237 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.603191 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:36.604085 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:36.605232 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:36.606431 18301 hybrid_clock.cc:648] HybridClock initialized: now 1790614116606414 us; error 34 us; skew 500 ppm
Sep 28 16:48:36 dist-test-slave-5rg1 krb5kdc[18288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614116, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:36.609337 18301 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:36.610380 18301 webserver.cc:460] Webserver started at http://127.30.196.62:42739/ using document root <none> and password file <none>
I20260928 16:48:36.611022 18301 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:36.611120 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:36.611362 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:36.613521 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "8061e390c94d430bbbda40286bca9473"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "bf16fba05af007d3b4ae211c6032869b"
server_key_iv: "6a529ce74cfc33fcef433ab27b569f1d"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.614089 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "8061e390c94d430bbbda40286bca9473"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "bf16fba05af007d3b4ae211c6032869b"
server_key_iv: "6a529ce74cfc33fcef433ab27b569f1d"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.617900 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:48:36.620393 18316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.621652 18301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:36.621791 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "8061e390c94d430bbbda40286bca9473"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "bf16fba05af007d3b4ae211c6032869b"
server_key_iv: "6a529ce74cfc33fcef433ab27b569f1d"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.621907 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:36.632054 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:36.634788 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:36.634994 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:36.642308 18301 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:33141
I20260928 16:48:36.642320 18368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:33141 every 8 connection(s)
I20260928 16:48:36.643270 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:36.646113 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:36.651770 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: Bootstrap starting.
I20260928 16:48:36.653596 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18301
I20260928 16:48:36.653730 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:36.654039 31504 external_mini_cluster.cc:1428] Setting key 953cd18a70da2df99e840b364a18acb1
I20260928 16:48:36.654560 18369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:36.655416 18369 log.cc:824] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:36.657419 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: No bootstrap required, opened a new log
I20260928 16:48:36.660956 18369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } }
I20260928 16:48:36.661232 18369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:36.661293 18369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8061e390c94d430bbbda40286bca9473, State: Initialized, Role: FOLLOWER
I20260928 16:48:36.661744 18369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } }
I20260928 16:48:36.661872 18369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:36.661968 18369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:36.662091 18369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:36.663192 18369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } }
Sep 28 16:48:36 dist-test-slave-5rg1 krb5kdc[18288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:36.663558 18369 leader_election.cc:302] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8061e390c94d430bbbda40286bca9473; no voters:
I20260928 16:48:36.663937 18369 leader_election.cc:288] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:36.664012 18374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:36.664224 18374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [term 1 LEADER]: Becoming Leader. State: Replica: 8061e390c94d430bbbda40286bca9473, State: Running, Role: LEADER
I20260928 16:48:36.664533 18374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } }
I20260928 16:48:36.665277 18369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:36.666167 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8061e390c94d430bbbda40286bca9473. Latest consensus state: current_term: 1 leader_uuid: "8061e390c94d430bbbda40286bca9473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } } }
I20260928 16:48:36.666306 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:36.666666 18376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8061e390c94d430bbbda40286bca9473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8061e390c94d430bbbda40286bca9473" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 33141 } } }
I20260928 16:48:36.666798 18376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:36.667933 18383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:36.671092 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.655328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.30.196.62:33141)
0928 16:48:36.655962 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.655984 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.656029 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:48:36.656758 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.656776 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.657115 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.657458 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:36.658275 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.659262 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.660208 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.660699 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.660894 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.664022 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:36.664031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:36.664052 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:36.664090 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:36.667955 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:36.668495 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:36.668500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:36.668505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:36.668616 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:36.668913 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:36.668915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:36.668917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:36.669331 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:36.669492 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.669846 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.670155 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":158,"threads_started":1}
I20260928 16:48:36.671284 18383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:36.677033 18383 catalog_manager.cc:1347] Generated new cluster ID: 963554abc590423ea4b06b54534df2cf
I20260928 16:48:36.677121 18383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:36.691510 18383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:36.692497 18383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:36.700215 18383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: Generated new TSK 0
I20260928 16:48:36.700896 18383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.196.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
Entry for principal kudu/127.30.196.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab.
I20260928 16:48:36.731051 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:36.845683 18395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:36.845957 18395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:36.846001 18395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:36.850289 18395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:36.850365 18395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:36.850477 18395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:36.856158 18395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.1.keytab
--principal=kudu/127.30.196.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:36.857417 18395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:36.858500 18395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:36.868542 18400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.868542 18401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.868598 18403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:36.868710 18395 server_base.cc:1034] running on GCE node
I20260928 16:48:36.869184 18395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:36.869887 18395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:36.871078 18395 hybrid_clock.cc:648] HybridClock initialized: now 1790614116871065 us; error 29 us; skew 500 ppm
Sep 28 16:48:36 dist-test-slave-5rg1 krb5kdc[18288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614116, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:36.873875 18395 init.cc:377] Logged in from keytab as kudu/127.30.196.1@KRBTEST.COM (short username kudu)
I20260928 16:48:36.874782 18395 webserver.cc:460] Webserver started at http://127.30.196.1:37575/ using document root <none> and password file <none>
I20260928 16:48:36.875370 18395 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:36.875471 18395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:36.875763 18395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:36.877892 18395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "3b52921e5aaf46f29f28af7ab185b827"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "e61f36787f4395edb5ab8bd52f67f818"
server_key_iv: "e62d71c10991e7dc1c753b2e248d12f2"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.878513 18395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "3b52921e5aaf46f29f28af7ab185b827"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "e61f36787f4395edb5ab8bd52f67f818"
server_key_iv: "e62d71c10991e7dc1c753b2e248d12f2"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.882376 18395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:36.884821 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.886027 18395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:36.886170 18395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "3b52921e5aaf46f29f28af7ab185b827"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-5rg1"
server_key: "e61f36787f4395edb5ab8bd52f67f818"
server_key_iv: "e62d71c10991e7dc1c753b2e248d12f2"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.886283 18395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:36.900036 18395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:36.902882 18395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:36.903100 18395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:36.903753 18395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:36.904606 18395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:36.904677 18395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.904750 18395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:36.904794 18395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.914160 18395 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:40081
I20260928 16:48:36.914183 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:40081 every 8 connection(s)
I20260928 16:48:36.915060 18395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:36.917758 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18395
I20260928 16:48:36.917896 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:36.918241 31504 external_mini_cluster.cc:1428] Setting key cc351c525569bfc79f81a1ff054dd232
Sep 28 16:48:36 dist-test-slave-5rg1 krb5kdc[18288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614116, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.1@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.2@KRBTEST.COM; defaulting to no policy
I20260928 16:48:36.929811 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.916931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:47425 (local address 127.30.196.62:33141)
0928 16:48:36.917067 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.917070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.918633 (+ 1563us) server_negotiation.cc:407] Connection header received
0928 16:48:36.920125 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.920128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.920182 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.920265 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:36.921426 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.921969 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.923107 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.923345 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.923473 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.926153 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:36.926158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:36.926163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:36.926194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:36.927847 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:36.928353 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:36.928360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:36.928365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:36.928466 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:36.928765 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:36.928767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:36.928768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:36.928956 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:36.929032 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.929522 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.929672 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.2@KRBTEST.COM" created.
I20260928 16:48:36.930706 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.917361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33141 (local address 127.30.196.1:47425)
0928 16:48:36.918483 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.918525 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.919923 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.920447 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.920454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.920805 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:36.921275 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.921288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.922107 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.922111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.922979 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.922989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.923518 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.923522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.923750 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.924375 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:36.924392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:36.925924 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:36.927954 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:36.927958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:36.927987 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:36.928238 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:36.928547 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:36.928549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:36.928550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:36.928685 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:36.929045 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:36.929050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:36.929284 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.929440 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.929756 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":157,"threads_started":1}
I20260928 16:48:36.932080 18526 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33141
I20260928 16:48:36.932338 18526 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:36.932837 18526 heartbeater.cc:511] Master 127.30.196.62:33141 requested a full tablet report, sending...
I20260928 16:48:36.934406 18333 ts_manager.cc:194] Registered new tserver with Master: 3b52921e5aaf46f29f28af7ab185b827 (127.30.196.1:40081)
I20260928 16:48:36.935536 18333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.1@KRBTEST.COM'} at 127.30.196.1:47425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
Entry for principal kudu/127.30.196.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab.
I20260928 16:48:36.946981 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:37.059343 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:37.059619 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:37.059722 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:37.063844 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:37.063905 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:37.063962 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:37.069443 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.2.keytab
--principal=kudu/127.30.196.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:37.070489 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:37.071465 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:37.081336 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.081339 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.081370 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:37.081576 18531 server_base.cc:1034] running on GCE node
I20260928 16:48:37.082113 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:37.082779 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:37.083992 18531 hybrid_clock.cc:648] HybridClock initialized: now 1790614117083969 us; error 34 us; skew 500 ppm
Sep 28 16:48:37 dist-test-slave-5rg1 krb5kdc[18288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:37.086587 18531 init.cc:377] Logged in from keytab as kudu/127.30.196.2@KRBTEST.COM (short username kudu)
I20260928 16:48:37.087491 18531 webserver.cc:460] Webserver started at http://127.30.196.2:36883/ using document root <none> and password file <none>
I20260928 16:48:37.088137 18531 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:37.088233 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:37.088451 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:37.090678 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "0e5ca74921774eafb244cf8b290271f7"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "0006c19d8c81c2806691b12cc7ac4017"
server_key_iv: "cbb4db2a76f96cfdaa0209c6736048c0"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.091244 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "0e5ca74921774eafb244cf8b290271f7"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "0006c19d8c81c2806691b12cc7ac4017"
server_key_iv: "cbb4db2a76f96cfdaa0209c6736048c0"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.095119 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:37.097695 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.098986 18531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:37.099143 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "0e5ca74921774eafb244cf8b290271f7"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "0006c19d8c81c2806691b12cc7ac4017"
server_key_iv: "cbb4db2a76f96cfdaa0209c6736048c0"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.099274 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:37.118024 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:37.121394 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:37.121625 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:37.122247 18531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:37.123123 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:37.123198 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.123270 18531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:37.123323 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.132107 18531 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:40557
I20260928 16:48:37.132181 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:40557 every 8 connection(s)
I20260928 16:48:37.133029 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:37.133926 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18531
I20260928 16:48:37.134109 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:37.134405 31504 external_mini_cluster.cc:1428] Setting key 2a2cebb7a6abe8aa4cbb9b06ed866a3d
WARNING: no policy specified for kudu/127.30.196.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.3@KRBTEST.COM" created.
Sep 28 16:48:37 dist-test-slave-5rg1 krb5kdc[18288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.2@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:37.152307 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.136275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:48959 (local address 127.30.196.62:33141)
0928 16:48:37.136429 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.136434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.137590 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 16:48:37.139023 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.139029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.139092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.139188 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.140570 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.141137 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.142314 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.142552 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.142667 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.148329 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.148332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.148334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:37.148362 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.150014 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.150624 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.150628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.150630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.150716 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.151074 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.151078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.151080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.151281 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.151394 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.152000 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.152134 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:48:37.153224 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.136790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33141 (local address 127.30.196.2:48959)
0928 16:48:37.137419 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.137475 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.138730 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.139375 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.139382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.139868 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:37.140413 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.140428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.141279 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.141283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.142180 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.142189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.142712 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.142716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.142891 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.143532 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:37.143549 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:37.148107 (+ 4558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:37.150166 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:37.150173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:37.150197 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:37.150508 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:37.150816 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:37.150820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:37.150822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:37.150974 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:37.151404 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:37.151414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:37.151743 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.151955 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.152326 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":175,"threads_started":1}
I20260928 16:48:37.154711 18662 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33141
I20260928 16:48:37.155043 18662 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:37.155568 18662 heartbeater.cc:511] Master 127.30.196.62:33141 requested a full tablet report, sending...
I20260928 16:48:37.156740 18333 ts_manager.cc:194] Registered new tserver with Master: 0e5ca74921774eafb244cf8b290271f7 (127.30.196.2:40557)
I20260928 16:48:37.157366 18333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.2@KRBTEST.COM'} at 127.30.196.2:48959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
Entry for principal kudu/127.30.196.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab.
I20260928 16:48:37.166989 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:37.282266 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:37.282608 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:37.282694 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:37.286890 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:37.286971 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:37.287034 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:37.292972 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.3.keytab
--principal=kudu/127.30.196.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:37.294051 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:37.295063 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:48:37.305295 18667 server_base.cc:1034] running on GCE node
W20260928 16:48:37.305240 18673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.305240 18675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.305238 18672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:37.306274 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:37.307029 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:37.308292 18667 hybrid_clock.cc:648] HybridClock initialized: now 1790614117308272 us; error 32 us; skew 500 ppm
Sep 28 16:48:37 dist-test-slave-5rg1 krb5kdc[18288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:37.311226 18667 init.cc:377] Logged in from keytab as kudu/127.30.196.3@KRBTEST.COM (short username kudu)
I20260928 16:48:37.312196 18667 webserver.cc:460] Webserver started at http://127.30.196.3:39535/ using document root <none> and password file <none>
I20260928 16:48:37.312737 18667 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:37.312798 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:37.312978 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:37.314975 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "dd4e3ce39db64a959ef5145a09dea853"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "cd3aec01a84a44288fba832dcbdc5789"
server_key_iv: "f4dab012fb20187b36ab682e458a2d1a"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.315562 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4e3ce39db64a959ef5145a09dea853"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "cd3aec01a84a44288fba832dcbdc5789"
server_key_iv: "f4dab012fb20187b36ab682e458a2d1a"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.319422 18667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:48:37.322597 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.324098 18667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:37.324239 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "dd4e3ce39db64a959ef5145a09dea853"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "cd3aec01a84a44288fba832dcbdc5789"
server_key_iv: "f4dab012fb20187b36ab682e458a2d1a"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.324353 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:37.342856 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:37.345886 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:37.346118 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:37.346728 18667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:37.347579 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:37.347649 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.347739 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:37.347786 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.356674 18667 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:35107
I20260928 16:48:37.356696 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:35107 every 8 connection(s)
I20260928 16:48:37.357573 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:37.364331 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18667
I20260928 16:48:37.364466 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:37.364754 31504 external_mini_cluster.cc:1428] Setting key e710c62b82606e02a590a907e1f67da3
Sep 28 16:48:37 dist-test-slave-5rg1 krb5kdc[18288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.3@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:37.371876 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.359410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:48723 (local address 127.30.196.62:33141)
0928 16:48:37.359592 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.359598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.360490 (+ 892us) server_negotiation.cc:407] Connection header received
0928 16:48:37.361759 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.361761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.361812 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.361889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.363051 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.363585 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.365261 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.365553 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.365678 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.368600 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.368603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.368605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:37.368631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.369950 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.370443 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.370445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.370447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.370527 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.370825 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.370827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.370828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.371015 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.371089 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.371574 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.371744 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:48:37.372697 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.359869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:33141 (local address 127.30.196.3:48723)
0928 16:48:37.360326 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.360370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.361560 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.362057 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.362064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.362423 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:37.362921 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.362934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.363733 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.363735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.365122 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.365134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.365750 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.365758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.365974 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.366663 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:37.366680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:37.368430 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:37.370056 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:37.370062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:37.370075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:37.370348 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:37.370603 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:37.370605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:37.370606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:37.370739 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:37.371097 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:37.371102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:37.371358 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.371510 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.371845 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260928 16:48:37.373782 18797 heartbeater.cc:347] Connected to a master server at 127.30.196.62:33141
I20260928 16:48:37.374053 18797 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:37.374554 18797 heartbeater.cc:511] Master 127.30.196.62:33141 requested a full tablet report, sending...
I20260928 16:48:37.375608 18333 ts_manager.cc:194] Registered new tserver with Master: dd4e3ce39db64a959ef5145a09dea853 (127.30.196.3:35107)
I20260928 16:48:37.376212 18333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.196.3@KRBTEST.COM'} at 127.30.196.3:48723
I20260928 16:48:37.378801 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:37.387863 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.380789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.30.196.62:33141)
0928 16:48:37.380934 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.380940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.381232 (+ 292us) server_negotiation.cc:407] Connection header received
0928 16:48:37.381318 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.381320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.381360 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.381423 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.382050 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.382552 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.383333 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.383544 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.383617 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.384918 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.384921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.384923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:37.384944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.386342 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.386693 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.386695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.386696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.386767 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:37.387007 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:37.387010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:37.387011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:37.387217 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.387319 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.387556 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.387698 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:48:37.390827 18333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:37.393998 18333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:37.408706 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.402475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40081 (local address 127.0.0.1:58790)
0928 16:48:37.403548 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.403560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.403663 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.404613 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.404617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.404638 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.404745 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.404749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.406097 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.406102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.407184 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.407194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408141 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.408252 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.408287 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.408399 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.408540 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":59,"thread_start_us":100,"threads_started":1}
I20260928 16:48:37.408767 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.402614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.30.196.1:40081)
0928 16:48:37.403912 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.403929 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.403998 (+ 69us) server_negotiation.cc:407] Connection header received
0928 16:48:37.404089 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.404093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.404290 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.404438 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.404867 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.405961 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.407316 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408017 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408140 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.408293 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.408538 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.408646 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":118,"threads_started":1}
I20260928 16:48:37.409175 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.403249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.30.196.2:40557)
0928 16:48:37.403757 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.403774 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.404008 (+ 234us) server_negotiation.cc:407] Connection header received
0928 16:48:37.404209 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.404214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.404365 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.404503 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.404838 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.405580 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.407268 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408121 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408230 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.408371 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.408850 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.409037 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":123,"threads_started":1}
I20260928 16:48:37.408739 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.402840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:40557 (local address 127.0.0.1:59652)
0928 16:48:37.403909 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.403922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.404074 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.404542 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.404545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.404560 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.404709 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.404720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.405720 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.405724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.407129 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.407140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408372 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.408492 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.408513 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.408577 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.408635 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":43,"thread_start_us":82,"threads_started":1}
I20260928 16:48:37.409790 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.403295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.30.196.3:35107)
0928 16:48:37.403845 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.403857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.403880 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:48:37.403962 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.403968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.404194 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.404352 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.404670 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.405905 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.407494 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408303 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408408 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.408553 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.409545 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.409642 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":107,"threads_started":1}
I20260928 16:48:37.410123 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.402523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:35107 (local address 127.0.0.1:52130)
0928 16:48:37.403144 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.403196 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.403324 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.404352 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.404358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.404380 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.404541 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.404548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.406195 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.406201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.407363 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.407372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.408449 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.408453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.409084 (+ 631us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.409108 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.409182 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.409290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":542,"thread_start_us":119,"threads_started":1}
I20260928 16:48:37.412515 18730 tablet_service.cc:1467] Processing CreateTablet for tablet bfd56f48897f41afbc1190507dc3ec44 (DEFAULT_TABLE table=test-table [id=e591e7ad74254cd6a10c46c554858065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:37.412456 18458 tablet_service.cc:1467] Processing CreateTablet for tablet bfd56f48897f41afbc1190507dc3ec44 (DEFAULT_TABLE table=test-table [id=e591e7ad74254cd6a10c46c554858065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:37.412456 18595 tablet_service.cc:1467] Processing CreateTablet for tablet bfd56f48897f41afbc1190507dc3ec44 (DEFAULT_TABLE table=test-table [id=e591e7ad74254cd6a10c46c554858065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:37.413525 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd56f48897f41afbc1190507dc3ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:37.413525 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd56f48897f41afbc1190507dc3ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:37.413545 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd56f48897f41afbc1190507dc3ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:37.419824 18814 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Bootstrap starting.
I20260928 16:48:37.420863 18815 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: Bootstrap starting.
I20260928 16:48:37.422346 18814 tablet_bootstrap.cc:654] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:37.422840 18815 tablet_bootstrap.cc:654] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:37.423426 18814 log.cc:824] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:37.423519 18815 log.cc:824] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:37.423964 18816 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: Bootstrap starting.
I20260928 16:48:37.425578 18815 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: No bootstrap required, opened a new log
I20260928 16:48:37.425776 18815 ts_tablet_manager.cc:1397] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:48:37.426105 18814 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: No bootstrap required, opened a new log
I20260928 16:48:37.426245 18816 tablet_bootstrap.cc:654] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:37.426291 18814 ts_tablet_manager.cc:1397] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:48:37.427480 18816 log.cc:824] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:37.429392 18815 raft_consensus.cc:348] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.429549 18816 tablet_bootstrap.cc:492] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: No bootstrap required, opened a new log
I20260928 16:48:37.429641 18815 raft_consensus.cc:374] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:37.429702 18815 raft_consensus.cc:729] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5ca74921774eafb244cf8b290271f7, State: Initialized, Role: FOLLOWER
I20260928 16:48:37.429740 18816 ts_tablet_manager.cc:1397] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:48:37.430207 18815 consensus_queue.cc:260] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.430163 18814 raft_consensus.cc:348] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.430416 18814 raft_consensus.cc:374] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:37.430490 18814 raft_consensus.cc:729] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4e3ce39db64a959ef5145a09dea853, State: Initialized, Role: FOLLOWER
I20260928 16:48:37.430943 18814 consensus_queue.cc:260] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.431219 18815 ts_tablet_manager.cc:1428] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:37.431597 18662 heartbeater.cc:503] Master 127.30.196.62:33141 was elected leader, sending a full tablet report...
I20260928 16:48:37.432158 18797 heartbeater.cc:503] Master 127.30.196.62:33141 was elected leader, sending a full tablet report...
I20260928 16:48:37.432476 18814 ts_tablet_manager.cc:1428] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:48:37.433403 18816 raft_consensus.cc:348] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.433617 18816 raft_consensus.cc:374] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:37.433691 18816 raft_consensus.cc:729] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52921e5aaf46f29f28af7ab185b827, State: Initialized, Role: FOLLOWER
I20260928 16:48:37.434094 18816 consensus_queue.cc:260] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.434679 18526 heartbeater.cc:503] Master 127.30.196.62:33141 was elected leader, sending a full tablet report...
I20260928 16:48:37.434958 18816 ts_tablet_manager.cc:1428] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:37.453398 18820 raft_consensus.cc:482] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:37.453646 18820 raft_consensus.cc:504] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.454893 18820 leader_election.cc:288] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b52921e5aaf46f29f28af7ab185b827 (127.30.196.1:40081), dd4e3ce39db64a959ef5145a09dea853 (127.30.196.3:35107)
I20260928 16:48:37.458887 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.455374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:42415 (local address 127.30.196.3:35107)
0928 16:48:37.455524 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.455529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.455739 (+ 210us) server_negotiation.cc:407] Connection header received
0928 16:48:37.455856 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.455859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.455911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.455995 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.456296 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.456832 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.457894 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.458321 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.458436 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.458501 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.458707 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.458775 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:48:37.458887 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.455107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40081 (local address 127.30.196.2:33221)
0928 16:48:37.455310 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.455325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.455421 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.455778 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.455781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.455806 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.455946 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.455966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.456787 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.456790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.457698 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.457707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.458380 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.458385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.458533 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.458559 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.458625 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.458737 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":27}
I20260928 16:48:37.458889 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.455213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:33221 (local address 127.30.196.1:40081)
0928 16:48:37.455383 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.455389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.455410 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:37.455564 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.455567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.455629 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.455762 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.456076 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.456676 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.457835 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.458255 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.458351 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.458432 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.458673 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.458753 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:48:37.458909 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.455189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:35107 (local address 127.30.196.2:42415)
0928 16:48:37.455620 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.455634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.455754 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.456035 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.456040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.456060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.456183 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.456189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.456946 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.456949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.457730 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.457739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.458454 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.458459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.458560 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.458577 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.458671 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.458772 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":23,"thread_start_us":166,"threads_started":1}
I20260928 16:48:37.459625 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd56f48897f41afbc1190507dc3ec44" candidate_uuid: "0e5ca74921774eafb244cf8b290271f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52921e5aaf46f29f28af7ab185b827" is_pre_election: true
I20260928 16:48:37.459780 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd56f48897f41afbc1190507dc3ec44" candidate_uuid: "0e5ca74921774eafb244cf8b290271f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4e3ce39db64a959ef5145a09dea853" is_pre_election: true
I20260928 16:48:37.460014 18750 raft_consensus.cc:2454] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5ca74921774eafb244cf8b290271f7 in term 0.
I20260928 16:48:37.460012 18479 raft_consensus.cc:2454] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5ca74921774eafb244cf8b290271f7 in term 0.
I20260928 16:48:37.460517 18549 leader_election.cc:302] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5ca74921774eafb244cf8b290271f7, dd4e3ce39db64a959ef5145a09dea853; no voters:
I20260928 16:48:37.460757 18820 raft_consensus.cc:2781] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:37.460857 18820 raft_consensus.cc:482] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:37.460920 18820 raft_consensus.cc:3037] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.461992 18820 raft_consensus.cc:504] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.462386 18820 leader_election.cc:288] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [CANDIDATE]: Term 1 election: Requested vote from peers 3b52921e5aaf46f29f28af7ab185b827 (127.30.196.1:40081), dd4e3ce39db64a959ef5145a09dea853 (127.30.196.3:35107)
I20260928 16:48:37.462702 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd56f48897f41afbc1190507dc3ec44" candidate_uuid: "0e5ca74921774eafb244cf8b290271f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52921e5aaf46f29f28af7ab185b827"
I20260928 16:48:37.462809 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd56f48897f41afbc1190507dc3ec44" candidate_uuid: "0e5ca74921774eafb244cf8b290271f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4e3ce39db64a959ef5145a09dea853"
I20260928 16:48:37.462836 18479 raft_consensus.cc:3037] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.462926 18750 raft_consensus.cc:3037] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.464026 18479 raft_consensus.cc:2454] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5ca74921774eafb244cf8b290271f7 in term 1.
I20260928 16:48:37.464068 18750 raft_consensus.cc:2454] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5ca74921774eafb244cf8b290271f7 in term 1.
I20260928 16:48:37.464340 18547 leader_election.cc:302] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5ca74921774eafb244cf8b290271f7, 3b52921e5aaf46f29f28af7ab185b827; no voters:
I20260928 16:48:37.464543 18820 raft_consensus.cc:2781] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:37.464790 18820 raft_consensus.cc:686] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5ca74921774eafb244cf8b290271f7, State: Running, Role: LEADER
I20260928 16:48:37.465121 18820 consensus_queue.cc:237] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } }
I20260928 16:48:37.468509 18332 catalog_manager.cc:5495] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e5ca74921774eafb244cf8b290271f7 (127.30.196.2). New cstate: current_term: 1 leader_uuid: "0e5ca74921774eafb244cf8b290271f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5ca74921774eafb244cf8b290271f7" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 40557 } health_report { overall_health: HEALTHY } } }
I20260928 16:48:37.487264 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.483592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.30.196.2:40557)
0928 16:48:37.483759 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.483764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.483778 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:48:37.483940 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.483942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.483998 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.484111 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:37.484372 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.484882 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.485739 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.486010 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.486131 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.486436 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:37.486828 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:37.487013 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.487047 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.487103 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:48:37.493235 18479 raft_consensus.cc:1261] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 1 FOLLOWER]: Refusing update from remote peer 0e5ca74921774eafb244cf8b290271f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:37.493261 18750 raft_consensus.cc:1261] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 1 FOLLOWER]: Refusing update from remote peer 0e5ca74921774eafb244cf8b290271f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:37.494009 18824 consensus_queue.cc:1035] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52921e5aaf46f29f28af7ab185b827" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:37.494185 18820 consensus_queue.cc:1035] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4e3ce39db64a959ef5145a09dea853" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 35107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:37.511217 18332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:37.511399 18332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:37.513939 18332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8061e390c94d430bbbda40286bca9473: Sending DeleteTablet for 3 replicas of tablet bfd56f48897f41afbc1190507dc3ec44
I20260928 16:48:37.514743 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd56f48897f41afbc1190507dc3ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:58790
I20260928 16:48:37.514833 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd56f48897f41afbc1190507dc3ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:59652
I20260928 16:48:37.515128 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd56f48897f41afbc1190507dc3ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='kudu', principal='kudu/127.30.196.62@KRBTEST.COM'} at 127.0.0.1:52130
I20260928 16:48:37.515591 18837 tablet_replica.cc:330] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827: stopping tablet replica
I20260928 16:48:37.515964 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18395
I20260928 16:48:37.515957 18837 raft_consensus.cc:2229] T bfd56f48897f41afbc1190507dc3ec44 P 3b52921e5aaf46f29f28af7ab185b827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:37.524127 18838 tablet_replica.cc:330] T bfd56f48897f41afbc1190507dc3ec44 P 0e5ca74921774eafb244cf8b290271f7: stopping tablet replica
W20260928 16:48:37.524250 18317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.196.1:40081 (error 108)
I20260928 16:48:37.524307 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18531
I20260928 16:48:37.525449 18839 tablet_replica.cc:330] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: stopping tablet replica
I20260928 16:48:37.525833 18839 raft_consensus.cc:2229] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:37.526266 18839 raft_consensus.cc:2258] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:37.527405 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.527070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:40081 (local address 127.0.0.1:58806)
0928 16:48:37.527254 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.527320 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 16:48:37.527585 18317 catalog_manager.cc:4584] TS 3b52921e5aaf46f29f28af7ab185b827 (127.30.196.1:40081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfd56f48897f41afbc1190507dc3ec44: Network error: Client connection negotiation failed: client connection to 127.30.196.1:40081: connect: Connection refused (error 111)
I20260928 16:48:37.528030 18839 ts_tablet_manager.cc:1905] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:37.530635 18839 ts_tablet_manager.cc:1918] T bfd56f48897f41afbc1190507dc3ec44 P dd4e3ce39db64a959ef5145a09dea853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:37.530695 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18667
I20260928 16:48:37.532508 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.530897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:40557 (local address 127.0.0.1:59680)
0928 16:48:37.532367 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.532426 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.2:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1365}
I20260928 16:48:37.536481 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18301
2026-09-28T16:48:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2234 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T16:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:37Z Disabled control of system clock
I20260928 16:48:37.567492 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36949
--webserver_interface=127.30.196.62
--webserver_port=0
--builtin_ntp_servers=127.30.196.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:37.680573 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:37.680855 18844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:37.680891 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:37.685010 18844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:37.685077 18844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:37.685092 18844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:37.685109 18844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:37.685125 18844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:37.689970 18844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:37.691072 18844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:37.692454 18844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:37.701331 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.701381 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.701342 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:37.701413 18844 server_base.cc:1034] running on GCE node
I20260928 16:48:37.702100 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:37.703323 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:37.704488 18844 hybrid_clock.cc:648] HybridClock initialized: now 1790614117704453 us; error 52 us; skew 500 ppm
I20260928 16:48:37.706317 18844 webserver.cc:460] Webserver started at http://127.30.196.62:45609/ using document root <none> and password file <none>
I20260928 16:48:37.706903 18844 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:37.706967 18844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:37.707185 18844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:37.709354 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "5f3dea0db30b433396e831eaf306d57e"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "6ad0615a8a6e6a6d2868e133c4e40df5"
server_key_iv: "9dfef20b8e2efdce65525f5023797197"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.709962 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "5f3dea0db30b433396e831eaf306d57e"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "6ad0615a8a6e6a6d2868e133c4e40df5"
server_key_iv: "9dfef20b8e2efdce65525f5023797197"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.713755 18844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:48:37.716184 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.717401 18844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:37.717553 18844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "5f3dea0db30b433396e831eaf306d57e"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "6ad0615a8a6e6a6d2868e133c4e40df5"
server_key_iv: "9dfef20b8e2efdce65525f5023797197"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.717669 18844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:37.736272 18844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:37.737017 18844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:37.737229 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:37.744477 18844 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36949
I20260928 16:48:37.744499 18910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36949 every 8 connection(s)
I20260928 16:48:37.745428 18844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:37.748805 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:37.753988 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18844
I20260928 16:48:37.754148 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:37.754446 31504 external_mini_cluster.cc:1428] Setting key 40fa4b70a04440470242cb19eece27df
I20260928 16:48:37.754680 18911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: Bootstrap starting.
I20260928 16:48:37.757346 18911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:37.758119 18911 log.cc:824] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:37.760164 18911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: No bootstrap required, opened a new log
I20260928 16:48:37.764144 18911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } }
I20260928 16:48:37.764402 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.755924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.30.196.62:36949)
0928 16:48:37.756681 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.756697 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.756734 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:48:37.757524 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.757543 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.757949 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.758303 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.758786 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.759871 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.760869 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.761340 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.761528 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.762495 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.762500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.762513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:37.762909 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.762979 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:37.763000 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.763117 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.763276 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.763551 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":155,"threads_started":1}
I20260928 16:48:37.764425 18911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:37.764580 18911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3dea0db30b433396e831eaf306d57e, State: Initialized, Role: FOLLOWER
I20260928 16:48:37.765089 18911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } }
I20260928 16:48:37.765255 18911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:37.765341 18911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:37.765448 18911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.766429 18911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } }
I20260928 16:48:37.766784 18911 leader_election.cc:302] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f3dea0db30b433396e831eaf306d57e; no voters:
I20260928 16:48:37.767127 18911 leader_election.cc:288] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:37.767197 18916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:37.767509 18916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [term 1 LEADER]: Becoming Leader. State: Replica: 5f3dea0db30b433396e831eaf306d57e, State: Running, Role: LEADER
I20260928 16:48:37.767889 18916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } }
I20260928 16:48:37.768594 18911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:37.769696 18918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f3dea0db30b433396e831eaf306d57e. Latest consensus state: current_term: 1 leader_uuid: "5f3dea0db30b433396e831eaf306d57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } } }
I20260928 16:48:37.769836 18918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:37.770107 18917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f3dea0db30b433396e831eaf306d57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f3dea0db30b433396e831eaf306d57e" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36949 } } }
I20260928 16:48:37.770208 18917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e [sys.catalog]: This master's current role is: LEADER
W20260928 16:48:37.774008 18931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:48:37.774132 18931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:48:37.774256 18925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:37.774983 18925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:37.781155 18925 catalog_manager.cc:1347] Generated new cluster ID: 1f4710ee073646a4b42a152d2ccda717
I20260928 16:48:37.781242 18925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:37.796955 18925 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:37.797928 18925 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:37.805296 18925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: Generated new TSK 0
I20260928 16:48:37.806062 18925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:37.812923 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.1:0
--local_ip_for_outbound_sockets=127.30.196.1
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--builtin_ntp_servers=127.30.196.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:37.937145 18935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:37.937407 18935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:37.937445 18935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:37.941520 18935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:37.941573 18935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:37.941629 18935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.1
I20260928 16:48:37.946328 18935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.196.1
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.1
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:37.947407 18935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:37.948870 18935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:37.958827 18942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:37.958828 18944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:37.958884 18935 server_base.cc:1034] running on GCE node
W20260928 16:48:37.958827 18941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:37.959513 18935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:37.960256 18935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:37.961442 18935 hybrid_clock.cc:648] HybridClock initialized: now 1790614117961435 us; error 34 us; skew 500 ppm
I20260928 16:48:37.963202 18935 webserver.cc:460] Webserver started at http://127.30.196.1:42807/ using document root <none> and password file <none>
I20260928 16:48:37.963832 18935 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:37.963927 18935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:37.964165 18935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:37.966216 18935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data/instance:
uuid: "b50a24b194454bf7a0648f8a378d31f8"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "08ed1f1ab2ce0cec96156d6cd62583d2"
server_key_iv: "882b15b10c9cb3a18924fc011b16440e"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.966777 18935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance:
uuid: "b50a24b194454bf7a0648f8a378d31f8"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "08ed1f1ab2ce0cec96156d6cd62583d2"
server_key_iv: "882b15b10c9cb3a18924fc011b16440e"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.970705 18935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:37.973201 18950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:37.974422 18935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:37.974555 18935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal
uuid: "b50a24b194454bf7a0648f8a378d31f8"
format_stamp: "Formatted at 2026-09-28 16:48:37 on dist-test-slave-5rg1"
server_key: "08ed1f1ab2ce0cec96156d6cd62583d2"
server_key_iv: "882b15b10c9cb3a18924fc011b16440e"
server_key_version: "encryptionkey@0"
I20260928 16:48:37.974668 18935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:38.013932 18935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:38.014680 18935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:38.014887 18935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:38.015512 18935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:38.016507 18935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:38.016582 18935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.016657 18935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:38.016711 18935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.025921 18935 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.1:34225
I20260928 16:48:38.025951 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.1:34225 every 8 connection(s)
I20260928 16:48:38.026835 18935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:38.031533 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 18935
I20260928 16:48:38.031709 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:38.032001 31504 external_mini_cluster.cc:1428] Setting key 22c7353098e426c6bc3f4746fc0fa9f8
I20260928 16:48:38.033989 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.2:0
--local_ip_for_outbound_sockets=127.30.196.2
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--builtin_ntp_servers=127.30.196.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:38.036634 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.028767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.1:48155 (local address 127.30.196.62:36949)
0928 16:48:38.028913 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.028917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.029903 (+ 986us) server_negotiation.cc:407] Connection header received
0928 16:48:38.031307 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.031312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.031376 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.031484 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:38.032505 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.033224 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.034493 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.034887 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.035017 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.036117 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:38.036121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:38.036124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:38.036137 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:38.036165 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:38.036173 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:38.036252 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.036427 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.036491 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:48:38.037494 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.029137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36949 (local address 127.30.196.1:48155)
0928 16:48:38.029775 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.029815 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.031050 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.031654 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.031664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.031811 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:38.032352 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.032366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.033365 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.033369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.034371 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.034381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.035044 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.035046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.035184 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.035853 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:38.035854 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:38.035896 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:38.035962 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:38.036267 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:38.036279 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:38.036290 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.036412 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.036707 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260928 16:48:38.038740 19064 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36949
I20260928 16:48:38.038995 19064 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:38.039515 19064 heartbeater.cc:511] Master 127.30.196.62:36949 requested a full tablet report, sending...
I20260928 16:48:38.041260 18875 ts_manager.cc:194] Registered new tserver with Master: b50a24b194454bf7a0648f8a378d31f8 (127.30.196.1:34225)
I20260928 16:48:38.043045 18875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.1:48155
W20260928 16:48:38.147997 19067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:38.148274 19067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:38.148313 19067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:38.152503 19067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:38.152570 19067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:38.152626 19067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.2
I20260928 16:48:38.157765 19067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.196.2
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.2
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:38.158838 19067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:38.160341 19067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:38.170506 19075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:38.170521 19072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:38.170511 19073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:38.170677 19067 server_base.cc:1034] running on GCE node
I20260928 16:48:38.171164 19067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:38.171914 19067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:38.173084 19067 hybrid_clock.cc:648] HybridClock initialized: now 1790614118173051 us; error 44 us; skew 500 ppm
I20260928 16:48:38.174981 19067 webserver.cc:460] Webserver started at http://127.30.196.2:45589/ using document root <none> and password file <none>
I20260928 16:48:38.175601 19067 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:38.175776 19067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:38.176020 19067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:38.178159 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data/instance:
uuid: "31c7bd2fde764935bc03abbd2bfda8f0"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "c16df0e013a026ca6e6e93942789d78c"
server_key_iv: "1eac534b7bfd9b73335b0b0f34f8daef"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.178750 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance:
uuid: "31c7bd2fde764935bc03abbd2bfda8f0"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "c16df0e013a026ca6e6e93942789d78c"
server_key_iv: "1eac534b7bfd9b73335b0b0f34f8daef"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.182529 19067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:48:38.184978 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.186168 19067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:38.186290 19067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal
uuid: "31c7bd2fde764935bc03abbd2bfda8f0"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "c16df0e013a026ca6e6e93942789d78c"
server_key_iv: "1eac534b7bfd9b73335b0b0f34f8daef"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.186395 19067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:38.207940 19067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:38.208704 19067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:38.208932 19067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:38.209576 19067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:38.210561 19067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:38.210609 19067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.210675 19067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:38.210708 19067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.219308 19067 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.2:36211
I20260928 16:48:38.219328 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.2:36211 every 8 connection(s)
I20260928 16:48:38.220302 19067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:38.221129 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19067
I20260928 16:48:38.221276 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:38.221585 31504 external_mini_cluster.cc:1428] Setting key eb47daca398a0ce04444b9be0da3fda6
I20260928 16:48:38.223685 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.196.3:0
--local_ip_for_outbound_sockets=127.30.196.3
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--builtin_ntp_servers=127.30.196.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:38.231348 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.222338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.2:43181 (local address 127.30.196.62:36949)
0928 16:48:38.222496 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.222500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.223746 (+ 1246us) server_negotiation.cc:407] Connection header received
0928 16:48:38.225323 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.225325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.225371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.225442 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:38.226630 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.227154 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.229124 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.229339 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.229422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.230818 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:38.230822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:38.230825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:38.230839 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:38.230864 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:38.230875 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:38.230955 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.231162 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.231215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:48:38.232656 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.222825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36949 (local address 127.30.196.2:43181)
0928 16:48:38.223563 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.223615 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.225083 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.225632 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.225642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.225757 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:38.226460 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.226484 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.227296 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.227303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.228988 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.229002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.229518 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.229525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.229779 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.230449 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:38.230453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:38.230524 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:38.230621 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:38.230985 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:38.230997 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:38.231005 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.231152 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.231482 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
I20260928 16:48:38.233947 19196 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36949
I20260928 16:48:38.234210 19196 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:38.234740 19196 heartbeater.cc:511] Master 127.30.196.62:36949 requested a full tablet report, sending...
I20260928 16:48:38.235890 18875 ts_manager.cc:194] Registered new tserver with Master: 31c7bd2fde764935bc03abbd2bfda8f0 (127.30.196.2:36211)
I20260928 16:48:38.236465 18875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.2:43181
W20260928 16:48:38.341393 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:38.341689 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:38.341730 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:38.345911 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:38.345975 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:38.346053 19199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.196.3
I20260928 16:48:38.350924 19199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.196.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.196.3
--webserver_port=0
--tserver_master_addrs=127.30.196.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.196.3
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:38.352080 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:38.353469 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:38.363324 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:38.363512 19199 server_base.cc:1034] running on GCE node
W20260928 16:48:38.363360 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:38.363395 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:38.364050 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:38.364766 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:38.365955 19199 hybrid_clock.cc:648] HybridClock initialized: now 1790614118365946 us; error 34 us; skew 500 ppm
I20260928 16:48:38.367920 19199 webserver.cc:460] Webserver started at http://127.30.196.3:36417/ using document root <none> and password file <none>
I20260928 16:48:38.368508 19199 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:38.368573 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:38.368844 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:38.371094 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data/instance:
uuid: "9a3ae3c674254cc5b6b2cd61ee675a24"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "88f044d8be5e5c2166223a4ffef82d12"
server_key_iv: "97b5dd05cb75103e82de858799ec5ee3"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.371701 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance:
uuid: "9a3ae3c674254cc5b6b2cd61ee675a24"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "88f044d8be5e5c2166223a4ffef82d12"
server_key_iv: "97b5dd05cb75103e82de858799ec5ee3"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.375572 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:38.378063 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.379364 19199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:38.379490 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal
uuid: "9a3ae3c674254cc5b6b2cd61ee675a24"
format_stamp: "Formatted at 2026-09-28 16:48:38 on dist-test-slave-5rg1"
server_key: "88f044d8be5e5c2166223a4ffef82d12"
server_key_iv: "97b5dd05cb75103e82de858799ec5ee3"
server_key_version: "encryptionkey@0"
I20260928 16:48:38.379596 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:38.390762 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:38.391384 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:38.391570 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:38.392162 19199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:38.393061 19199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:38.393106 19199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.393173 19199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:38.393209 19199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:38.404585 19199 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.3:41021
I20260928 16:48:38.404808 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.3:41021 every 8 connection(s)
I20260928 16:48:38.405529 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:38.410953 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19199
I20260928 16:48:38.411101 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:38.411458 31504 external_mini_cluster.cc:1428] Setting key a2da6ef29474760b4c081065d4d20738
I20260928 16:48:38.423497 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.407656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:51633 (local address 127.30.196.62:36949)
0928 16:48:38.407842 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.407848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.409038 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 16:48:38.411187 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.411192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.411255 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.411357 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:38.412953 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.413681 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.415642 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.415986 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.416073 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.422965 (+ 6892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:38.422969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:38.422971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:38.422986 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:38.423010 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:38.423019 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:38.423094 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.423301 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.423369 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:48:38.425422 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.408048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:36949 (local address 127.30.196.3:51633)
0928 16:48:38.408851 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.408908 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.410877 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.411582 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.411593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.411753 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:38.412760 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.412783 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.413809 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.413814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.415504 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.415519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.416163 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.416169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.416414 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.422505 (+ 6091us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:38.422510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:38.422584 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:38.422699 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:38.423119 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:38.423132 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:38.423140 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.423280 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.424155 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260928 16:48:38.426590 19329 heartbeater.cc:347] Connected to a master server at 127.30.196.62:36949
I20260928 16:48:38.426913 19329 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:38.427647 19329 heartbeater.cc:511] Master 127.30.196.62:36949 requested a full tablet report, sending...
I20260928 16:48:38.428989 18875 ts_manager.cc:194] Registered new tserver with Master: 9a3ae3c674254cc5b6b2cd61ee675a24 (127.30.196.3:41021)
I20260928 16:48:38.429602 18875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.196.3:51633
I20260928 16:48:38.437358 31504 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:38.444137 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.439509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.30.196.62:36949)
0928 16:48:38.439694 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.439701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.439808 (+ 107us) server_negotiation.cc:407] Connection header received
0928 16:48:38.439918 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.439923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.439985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.440118 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:38.440350 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.441314 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.442199 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.442653 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.442781 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.443593 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:38.443598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:38.443603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:38.443620 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:38.443643 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:38.443656 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:38.443752 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.443856 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.443938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:48:38.447496 18875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:38.453213 18875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:38.470914 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.462813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:36211 (local address 127.0.0.1:44974)
0928 16:48:38.464654 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.464664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.464762 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.465392 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.465395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.465413 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:38.465525 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.465531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467211 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.467214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.469040 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.469051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.470363 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.470366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.470478 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.470519 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.470621 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.470739 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1792,"mutex_wait_us":241,"thread_start_us":105,"threads_started":1}
I20260928 16:48:38.471385 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.463357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.30.196.2:36211)
0928 16:48:38.464026 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.464039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.464752 (+ 713us) server_negotiation.cc:407] Connection header received
0928 16:48:38.464876 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.464882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.465076 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.465233 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:38.465655 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467090 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.469195 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.469995 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.470073 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.470261 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.471121 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.471223 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":129,"threads_started":1}
I20260928 16:48:38.470963 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.462543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.3:41021 (local address 127.0.0.1:42180)
0928 16:48:38.464266 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.464315 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.464462 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.465226 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.465235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.465266 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:38.465427 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.465437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467763 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.467769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.469475 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.469485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.470652 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.470655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.470732 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.470751 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.470799 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.470856 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"mutex_wait_us":217,"thread_start_us":78,"threads_started":1}
I20260928 16:48:38.472191 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.462541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:34225 (local address 127.0.0.1:36982)
0928 16:48:38.464928 (+ 2387us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.464940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.465076 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.465707 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.465711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.465727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:38.465796 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.465801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467211 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.467214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.469040 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.469051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.471605 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.471611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.471765 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.471802 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.471897 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.471994 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2331,"mutex_wait_us":812,"thread_start_us":92,"threads_started":1}
I20260928 16:48:38.472216 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.462805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.30.196.1:34225)
0928 16:48:38.463522 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.463537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.465014 (+ 1477us) server_negotiation.cc:407] Connection header received
0928 16:48:38.465224 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.465229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.465408 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.465527 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:38.465926 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467058 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.469171 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.471173 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.471286 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.471526 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.471954 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.472065 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":128,"threads_started":1}
I20260928 16:48:38.474864 19130 tablet_service.cc:1467] Processing CreateTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 (DEFAULT_TABLE table=test-table [id=98f468596e434e93917d0fb21bf719eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:38.475113 18998 tablet_service.cc:1467] Processing CreateTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 (DEFAULT_TABLE table=test-table [id=98f468596e434e93917d0fb21bf719eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:38.476054 18998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3337162b975a4f5a8bd77cf486c80ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:38.476054 19130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3337162b975a4f5a8bd77cf486c80ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:38.483633 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.463357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.30.196.3:41021)
0928 16:48:38.464083 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.464098 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.464441 (+ 343us) server_negotiation.cc:407] Connection header received
0928 16:48:38.464613 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.464622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.464852 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.465015 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:38.465586 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.467594 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.469612 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.470517 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.471315 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.483178 (+ 11863us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.483247 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.483388 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":152,"threads_started":1}
I20260928 16:48:38.484998 19347 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Bootstrap starting.
I20260928 16:48:38.486332 19260 tablet_service.cc:1467] Processing CreateTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 (DEFAULT_TABLE table=test-table [id=98f468596e434e93917d0fb21bf719eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:38.487226 19260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3337162b975a4f5a8bd77cf486c80ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:38.487805 19347 tablet_bootstrap.cc:654] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:38.488790 19347 log.cc:824] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:38.489609 19346 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Bootstrap starting.
I20260928 16:48:38.491106 19347 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: No bootstrap required, opened a new log
I20260928 16:48:38.491372 19347 ts_tablet_manager.cc:1397] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260928 16:48:38.492339 19346 tablet_bootstrap.cc:654] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:38.493278 19346 log.cc:824] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:38.495504 19346 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: No bootstrap required, opened a new log
I20260928 16:48:38.495733 19346 ts_tablet_manager.cc:1397] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:48:38.496157 19347 raft_consensus.cc:348] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.496522 19347 raft_consensus.cc:374] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:38.496629 19347 raft_consensus.cc:729] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c7bd2fde764935bc03abbd2bfda8f0, State: Initialized, Role: FOLLOWER
I20260928 16:48:38.496953 19350 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Bootstrap starting.
I20260928 16:48:38.497215 19347 consensus_queue.cc:260] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.498312 19347 ts_tablet_manager.cc:1428] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:48:38.498735 19196 heartbeater.cc:503] Master 127.30.196.62:36949 was elected leader, sending a full tablet report...
I20260928 16:48:38.499251 19350 tablet_bootstrap.cc:654] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:38.500046 19350 log.cc:824] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:38.502939 19346 raft_consensus.cc:348] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.503279 19346 raft_consensus.cc:374] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:38.503345 19346 raft_consensus.cc:729] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50a24b194454bf7a0648f8a378d31f8, State: Initialized, Role: FOLLOWER
I20260928 16:48:38.503952 19346 consensus_queue.cc:260] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.504745 19350 tablet_bootstrap.cc:492] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: No bootstrap required, opened a new log
I20260928 16:48:38.504733 19064 heartbeater.cc:503] Master 127.30.196.62:36949 was elected leader, sending a full tablet report...
I20260928 16:48:38.505069 19350 ts_tablet_manager.cc:1397] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 16:48:38.506342 19346 ts_tablet_manager.cc:1428] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:48:38.511483 19350 raft_consensus.cc:348] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.511823 19350 raft_consensus.cc:374] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:38.511905 19350 raft_consensus.cc:729] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ae3c674254cc5b6b2cd61ee675a24, State: Initialized, Role: FOLLOWER
I20260928 16:48:38.512415 19350 consensus_queue.cc:260] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.513230 19329 heartbeater.cc:503] Master 127.30.196.62:36949 was elected leader, sending a full tablet report...
I20260928 16:48:38.513584 19350 ts_tablet_manager.cc:1428] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260928 16:48:38.528110 19065 tablet.cc:2369] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:38.656648 19330 tablet.cc:2369] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:38.680954 19354 raft_consensus.cc:482] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:38.681192 19354 raft_consensus.cc:504] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.682412 19354 leader_election.cc:288] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b50a24b194454bf7a0648f8a378d31f8 (127.30.196.1:34225), 31c7bd2fde764935bc03abbd2bfda8f0 (127.30.196.2:36211)
I20260928 16:48:38.686897 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.682680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.2:36211 (local address 127.30.196.3:59501)
0928 16:48:38.682940 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.682965 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.683069 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.683353 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.683355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.683373 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:38.683499 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.683505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.684408 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.684414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.685840 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.685851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.686468 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.686472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.686576 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.686602 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.686680 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.686738 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":84}
I20260928 16:48:38.686923 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.682846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:59501 (local address 127.30.196.2:36211)
0928 16:48:38.682994 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.683001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.683019 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:48:38.683203 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.683205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.683248 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.683340 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:38.683614 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.684305 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.685962 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.686346 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.686441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.686521 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.686706 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.686796 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:48:38.687009 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.682811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.196.3:44571 (local address 127.30.196.1:34225)
0928 16:48:38.683016 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.683021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.683254 (+ 233us) server_negotiation.cc:407] Connection header received
0928 16:48:38.683409 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.683414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.683468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.683556 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:38.683846 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.684424 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.685998 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.686413 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.686540 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.686590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.686812 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.686880 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 16:48:38.687106 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.682697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.1:34225 (local address 127.30.196.3:44571)
0928 16:48:38.683157 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:38.683167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:38.683296 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:38.683568 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:38.683570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:38.683586 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:38.683737 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.683742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.684537 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.684540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.685842 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:38.685851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.686555 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.686559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:38.686684 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.686708 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:48:38.686784 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:48:38.686848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":37,"thread_start_us":145,"threads_started":1}
I20260928 16:48:38.687600 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3337162b975a4f5a8bd77cf486c80ac2" candidate_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" is_pre_election: true
I20260928 16:48:38.687637 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3337162b975a4f5a8bd77cf486c80ac2" candidate_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50a24b194454bf7a0648f8a378d31f8" is_pre_election: true
I20260928 16:48:38.687940 19150 raft_consensus.cc:2454] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ae3c674254cc5b6b2cd61ee675a24 in term 0.
I20260928 16:48:38.687966 19018 raft_consensus.cc:2454] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ae3c674254cc5b6b2cd61ee675a24 in term 0.
I20260928 16:48:38.688431 19214 leader_election.cc:302] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c7bd2fde764935bc03abbd2bfda8f0, 9a3ae3c674254cc5b6b2cd61ee675a24; no voters:
I20260928 16:48:38.688683 19354 raft_consensus.cc:2781] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:38.688766 19354 raft_consensus.cc:482] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:38.688822 19354 raft_consensus.cc:3037] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:38.689920 19354 raft_consensus.cc:504] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.690302 19354 leader_election.cc:288] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [CANDIDATE]: Term 1 election: Requested vote from peers b50a24b194454bf7a0648f8a378d31f8 (127.30.196.1:34225), 31c7bd2fde764935bc03abbd2bfda8f0 (127.30.196.2:36211)
I20260928 16:48:38.690690 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3337162b975a4f5a8bd77cf486c80ac2" candidate_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50a24b194454bf7a0648f8a378d31f8"
I20260928 16:48:38.690683 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3337162b975a4f5a8bd77cf486c80ac2" candidate_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c7bd2fde764935bc03abbd2bfda8f0"
I20260928 16:48:38.690845 19150 raft_consensus.cc:3037] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:38.690837 19018 raft_consensus.cc:3037] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:38.692085 19018 raft_consensus.cc:2454] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ae3c674254cc5b6b2cd61ee675a24 in term 1.
I20260928 16:48:38.692083 19150 raft_consensus.cc:2454] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ae3c674254cc5b6b2cd61ee675a24 in term 1.
I20260928 16:48:38.692448 19215 leader_election.cc:302] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3ae3c674254cc5b6b2cd61ee675a24, b50a24b194454bf7a0648f8a378d31f8; no voters:
I20260928 16:48:38.692672 19354 raft_consensus.cc:2781] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:38.692938 19354 raft_consensus.cc:686] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3ae3c674254cc5b6b2cd61ee675a24, State: Running, Role: LEADER
I20260928 16:48:38.693288 19354 consensus_queue.cc:237] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } }
I20260928 16:48:38.696292 18874 catalog_manager.cc:5495] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3ae3c674254cc5b6b2cd61ee675a24 (127.30.196.3). New cstate: current_term: 1 leader_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3ae3c674254cc5b6b2cd61ee675a24" member_type: VOTER last_known_addr { host: "127.30.196.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20260928 16:48:38.719880 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:38.716336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.30.196.3:41021)
0928 16:48:38.716481 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:38.716486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:38.716501 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:48:38.716625 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:38.716627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:38.716687 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:38.716805 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:38.717052 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.717553 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.718398 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:38.718618 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:38.718728 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:38.719035 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:38.719446 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:38.719579 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:38.719641 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:48:38.719718 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:48:38.721511 19197 tablet.cc:2369] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:38.725754 19150 raft_consensus.cc:1261] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 1 FOLLOWER]: Refusing update from remote peer 9a3ae3c674254cc5b6b2cd61ee675a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:38.725754 19018 raft_consensus.cc:1261] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 1 FOLLOWER]: Refusing update from remote peer 9a3ae3c674254cc5b6b2cd61ee675a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:38.726452 19354 consensus_queue.cc:1035] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c7bd2fde764935bc03abbd2bfda8f0" member_type: VOTER last_known_addr { host: "127.30.196.2" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:38.726639 19356 consensus_queue.cc:1035] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50a24b194454bf7a0648f8a378d31f8" member_type: VOTER last_known_addr { host: "127.30.196.1" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:38.736414 19360 mvcc.cc:205] Tried to move back new op lower bound from 7334355430293331968 to 7334355430170316800. Current Snapshot: MvccSnapshot[applied={T|T < 7334355430293331968}]
I20260928 16:48:38.742462 18874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:38.742604 18874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:38.745163 18874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f3dea0db30b433396e831eaf306d57e: Sending DeleteTablet for 3 replicas of tablet 3337162b975a4f5a8bd77cf486c80ac2
I20260928 16:48:38.746035 18998 tablet_service.cc:1514] Processing DeleteTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:38 UTC) from {username='slave'} at 127.0.0.1:36982
I20260928 16:48:38.746029 19130 tablet_service.cc:1514] Processing DeleteTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:38 UTC) from {username='slave'} at 127.0.0.1:44974
I20260928 16:48:38.746821 19260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3337162b975a4f5a8bd77cf486c80ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:38 UTC) from {username='slave'} at 127.0.0.1:42180
I20260928 16:48:38.747128 19368 tablet_replica.cc:330] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: stopping tablet replica
I20260928 16:48:38.747442 19368 raft_consensus.cc:2229] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:38.747787 19368 raft_consensus.cc:2258] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:38.748167 19369 tablet_replica.cc:330] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: stopping tablet replica
I20260928 16:48:38.748433 19369 raft_consensus.cc:2229] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:38.748683 19369 raft_consensus.cc:2258] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:38.750109 19368 ts_tablet_manager.cc:1905] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:38.750643 19369 ts_tablet_manager.cc:1905] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:38.752871 19368 ts_tablet_manager.cc:1918] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:38.752978 19368 log.cc:1196] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-0/wal/wals/3337162b975a4f5a8bd77cf486c80ac2
I20260928 16:48:38.753286 19368 ts_tablet_manager.cc:1939] T 3337162b975a4f5a8bd77cf486c80ac2 P b50a24b194454bf7a0648f8a378d31f8: Deleting consensus metadata
I20260928 16:48:38.753561 19369 ts_tablet_manager.cc:1918] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:38.753706 19369 log.cc:1196] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-1/wal/wals/3337162b975a4f5a8bd77cf486c80ac2
I20260928 16:48:38.754070 19369 ts_tablet_manager.cc:1939] T 3337162b975a4f5a8bd77cf486c80ac2 P 31c7bd2fde764935bc03abbd2bfda8f0: Deleting consensus metadata
I20260928 16:48:38.754549 18860 catalog_manager.cc:4841] TS b50a24b194454bf7a0648f8a378d31f8 (127.30.196.1:34225): tablet 3337162b975a4f5a8bd77cf486c80ac2 (table test-table [id=98f468596e434e93917d0fb21bf719eb]) successfully deleted
I20260928 16:48:38.755273 18859 catalog_manager.cc:4841] TS 31c7bd2fde764935bc03abbd2bfda8f0 (127.30.196.2:36211): tablet 3337162b975a4f5a8bd77cf486c80ac2 (table test-table [id=98f468596e434e93917d0fb21bf719eb]) successfully deleted
I20260928 16:48:38.755920 19370 tablet_replica.cc:330] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: stopping tablet replica
I20260928 16:48:38.756256 19370 raft_consensus.cc:2229] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:38.756387 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18935
I20260928 16:48:38.756577 19370 raft_consensus.cc:2258] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:38.759094 19370 ts_tablet_manager.cc:1905] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:38.761688 19370 ts_tablet_manager.cc:1918] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:38.761783 19370 log.cc:1196] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Deleting WAL directory at /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613859608463-31504-0/minicluster-data/ts-2/wal/wals/3337162b975a4f5a8bd77cf486c80ac2
I20260928 16:48:38.762087 19370 ts_tablet_manager.cc:1939] T 3337162b975a4f5a8bd77cf486c80ac2 P 9a3ae3c674254cc5b6b2cd61ee675a24: Deleting consensus metadata
I20260928 16:48:38.763213 18861 catalog_manager.cc:4841] TS 9a3ae3c674254cc5b6b2cd61ee675a24 (127.30.196.3:41021): tablet 3337162b975a4f5a8bd77cf486c80ac2 (table test-table [id=98f468596e434e93917d0fb21bf719eb]) successfully deleted
I20260928 16:48:38.765689 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19067
I20260928 16:48:38.773962 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19199
I20260928 16:48:38.780005 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 18844
2026-09-28T16:48:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1242 ms)
[----------] 6 tests from EncryptionPolicyTest (10675 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:38 dist-test-slave-5rg1 krb5kdc[19374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:38 dist-test-slave-5rg1 krb5kdc[19374](info): set up 2 sockets
Sep 28 16:48:38 dist-test-slave-5rg1 krb5kdc[19374](info): commencing operation
krb5kdc: starting...
W20260928 16:48:39.816365 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:39 dist-test-slave-5rg1 krb5kdc[19374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:39.947633 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:40.061026 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:40.061300 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:40.061342 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:40.065424 19387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:40.065481 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:40.065501 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:40.065517 19387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:40.065533 19387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:40.070787 19387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:40.071907 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:40.072975 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:40.081764 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:40.081785 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:40.081799 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:40.081825 19387 server_base.cc:1034] running on GCE node
I20260928 16:48:40.082579 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:40.083706 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:40.084906 19387 hybrid_clock.cc:648] HybridClock initialized: now 1790614120084886 us; error 30 us; skew 500 ppm
Sep 28 16:48:40 dist-test-slave-5rg1 krb5kdc[19374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:40.087779 19387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:40.088709 19387 webserver.cc:460] Webserver started at http://127.0.0.1:46163/ using document root <none> and password file <none>
I20260928 16:48:40.089274 19387 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:40.089366 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:40.089597 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:40.091607 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c"
format_stamp: "Formatted at 2026-09-28 16:48:40 on dist-test-slave-5rg1"
server_key: "a64b164b87bb73a64ffe4eff7488eb97"
server_key_iv: "c9361c2c3f54aea0f8ef89e45eb93639"
server_key_version: "encryptionkey@0"
I20260928 16:48:40.092226 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c"
format_stamp: "Formatted at 2026-09-28 16:48:40 on dist-test-slave-5rg1"
server_key: "a64b164b87bb73a64ffe4eff7488eb97"
server_key_iv: "c9361c2c3f54aea0f8ef89e45eb93639"
server_key_version: "encryptionkey@0"
I20260928 16:48:40.095942 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:40.098357 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:40.099632 19387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:40.099844 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c"
format_stamp: "Formatted at 2026-09-28 16:48:40 on dist-test-slave-5rg1"
server_key: "a64b164b87bb73a64ffe4eff7488eb97"
server_key_iv: "c9361c2c3f54aea0f8ef89e45eb93639"
server_key_version: "encryptionkey@0"
I20260928 16:48:40.099968 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:40.126091 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:40.129408 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:40.129639 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:40.137252 19387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38721
I20260928 16:48:40.137281 19455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38721 every 8 connection(s)
I20260928 16:48:40.138232 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:40.141090 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:40.144698 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19387
I20260928 16:48:40.144846 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:40.145164 31504 external_mini_cluster.cc:1428] Setting key 8c613c61ad91598c65d464d55ea2c1bd
I20260928 16:48:40.147377 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: Bootstrap starting.
I20260928 16:48:40.149642 19456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:40.150379 19456 log.cc:824] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:40.152281 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: No bootstrap required, opened a new log
Sep 28 16:48:40 dist-test-slave-5rg1 krb5kdc[19374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:40.156244 19456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260928 16:48:40.156464 19456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:40.156530 19456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c28fb9880d0438b92fd3b1f8f1ac53c, State: Initialized, Role: FOLLOWER
I20260928 16:48:40.156955 19456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260928 16:48:40.157071 19456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:40.157153 19456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:40.157244 19456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:40.158155 19456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260928 16:48:40.158557 19456 leader_election.cc:302] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c28fb9880d0438b92fd3b1f8f1ac53c; no voters:
I20260928 16:48:40.158954 19456 leader_election.cc:288] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:40.159209 19461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:40.159631 19461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [term 1 LEADER]: Becoming Leader. State: Replica: 3c28fb9880d0438b92fd3b1f8f1ac53c, State: Running, Role: LEADER
I20260928 16:48:40.159811 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:40.146576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35499 (local address 127.0.0.1:38721)
0928 16:48:40.147015 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:40.147029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:40.147063 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:48:40.147790 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:40.147807 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:40.148141 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:40.148461 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:40.149279 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:40.150335 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:40.151229 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:40.151755 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:40.151919 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:40.154617 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:40.154623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:40.154635 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:40.154663 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:40.156416 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:40.157017 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:40.157022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:40.157027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:40.157215 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:40.157583 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:40.157587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:40.157588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:40.157975 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:40.158202 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:40.158520 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:48:40.158751 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260928 16:48:40.160115 19461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260928 16:48:40.160403 19456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:40.161857 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260928 16:48:40.162067 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:40.162384 19463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c28fb9880d0438b92fd3b1f8f1ac53c. Latest consensus state: current_term: 1 leader_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c28fb9880d0438b92fd3b1f8f1ac53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260928 16:48:40.162472 19463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:40.163643 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 16:48:40.166138 19476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:48:40.166221 19476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:48:40.166302 19474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:40.167399 19474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:40.173749 19474 catalog_manager.cc:1347] Generated new cluster ID: 93a09000afaf4c858931f66454c59403
I20260928 16:48:40.173826 19474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:40.175614 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:40.165828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51089 (local address 127.0.0.1:38721)
0928 16:48:40.165958 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:40.165963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:40.166225 (+ 262us) server_negotiation.cc:407] Connection header received
0928 16:48:40.166435 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:40.166438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:40.166489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:40.166580 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:40.167496 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:40.168336 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:40.169568 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:40.169951 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:40.170058 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:40.171843 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:40.171847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:40.171852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:40.171882 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:40.173643 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:40.174227 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:40.174229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:40.174231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:40.174302 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:40.174674 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:40.174684 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:40.174687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:40.174861 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:40.175030 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:40.175317 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 16:48:40.175433 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:48:40.182125 19474 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:40.183422 19474 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:40.202592 19474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c28fb9880d0438b92fd3b1f8f1ac53c: Generated new TSK 0
I20260928 16:48:40.203572 19474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:40.217020 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19387
W20260928 16:48:40.222887 19371 connection.cc:535] client connection to 127.0.0.1:38721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38721): Connection reset by peer (error 104)
2026-09-28T16:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:40 dist-test-slave-5rg1 krb5kdc[19487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:40 dist-test-slave-5rg1 krb5kdc[19487](info): set up 2 sockets
Sep 28 16:48:40 dist-test-slave-5rg1 krb5kdc[19487](info): commencing operation
krb5kdc: starting...
W20260928 16:48:41.256340 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:41 dist-test-slave-5rg1 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:41.383744 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:41.494428 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:41.494750 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:41.494839 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:41.498848 19500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:41.498953 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:41.498992 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:41.499050 19500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:41.499091 19500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:41.504297 19500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:41.505375 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:41.506409 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:41.515537 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:41.515740 19500 server_base.cc:1034] running on GCE node
W20260928 16:48:41.515585 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:41.515537 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:41.516569 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:41.517757 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:41.518975 19500 hybrid_clock.cc:648] HybridClock initialized: now 1790614121518954 us; error 33 us; skew 500 ppm
Sep 28 16:48:41 dist-test-slave-5rg1 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:41.521831 19500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:41.522805 19500 webserver.cc:460] Webserver started at http://127.0.0.1:44285/ using document root <none> and password file <none>
I20260928 16:48:41.523357 19500 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:41.523418 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:41.523712 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:41.525864 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "fdcc51a527d448918e18ec17a25c1bd8"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-5rg1"
server_key: "15aecf99fb523393e11234848bf0bbdd"
server_key_iv: "153144c1ad3e3c8b99e6838eb5531031"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.526464 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "fdcc51a527d448918e18ec17a25c1bd8"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-5rg1"
server_key: "15aecf99fb523393e11234848bf0bbdd"
server_key_iv: "153144c1ad3e3c8b99e6838eb5531031"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.530504 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:48:41.532977 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:41.534297 19500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:41.534485 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "fdcc51a527d448918e18ec17a25c1bd8"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-5rg1"
server_key: "15aecf99fb523393e11234848bf0bbdd"
server_key_iv: "153144c1ad3e3c8b99e6838eb5531031"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.534606 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:41.554127 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:41.557439 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:41.557649 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:41.565251 19500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42911
I20260928 16:48:41.565267 19567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42911 every 8 connection(s)
I20260928 16:48:41.566207 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:41.569103 19568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:41.570647 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19500
I20260928 16:48:41.570813 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:41.571116 31504 external_mini_cluster.cc:1428] Setting key 3f84e5b3d17819b9cb381eaea1da91f7
I20260928 16:48:41.575359 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8: Bootstrap starting.
I20260928 16:48:41.578274 19568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:41.579293 19568 log.cc:824] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:41.580423 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:41.572615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33993 (local address 127.0.0.1:42911)
0928 16:48:41.573032 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:41.573047 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:41.573082 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:48:41.573710 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:41.573727 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:41.574064 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:41.574328 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:41.574752 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.575868 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.577078 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.577530 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.577649 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:41.578574 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:41.578578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:41.578591 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:41.579015 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:41.579074 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:41.579097 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:41.579201 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:41.579502 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:48:41.579646 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260928 16:48:41.581513 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8: No bootstrap required, opened a new log
I20260928 16:48:41.583488 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:41.586104 19568 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } }
I20260928 16:48:41.586467 19568 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:41.586548 19568 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcc51a527d448918e18ec17a25c1bd8, State: Initialized, Role: FOLLOWER
I20260928 16:48:41.587070 19568 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } }
I20260928 16:48:41.587248 19568 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:41.587320 19568 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:41.587451 19568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:41.588717 19568 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } }
I20260928 16:48:41.589134 19568 leader_election.cc:302] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdcc51a527d448918e18ec17a25c1bd8; no voters:
I20260928 16:48:41.589500 19568 leader_election.cc:288] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:41.589705 19578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:41.589923 19578 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [term 1 LEADER]: Becoming Leader. State: Replica: fdcc51a527d448918e18ec17a25c1bd8, State: Running, Role: LEADER
I20260928 16:48:41.590243 19578 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } }
I20260928 16:48:41.590478 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:41.585640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58719 (local address 127.0.0.1:42911)
0928 16:48:41.585754 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:41.585758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:41.585990 (+ 232us) server_negotiation.cc:407] Connection header received
0928 16:48:41.586064 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:41.586067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:41.586124 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:41.586219 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:41.586459 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.587288 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.588273 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.588736 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.588872 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:41.589850 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:41.589854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:41.589857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:41.589872 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:41.589903 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:41.589918 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:41.590021 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:41.590316 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 16:48:41.590355 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 16:48:41.591120 19568 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:41.591496 19580 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcc51a527d448918e18ec17a25c1bd8. Latest consensus state: current_term: 1 leader_uuid: "fdcc51a527d448918e18ec17a25c1bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } } }
I20260928 16:48:41.591631 19580 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:41.591959 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdcc51a527d448918e18ec17a25c1bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcc51a527d448918e18ec17a25c1bd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42911 } } }
I20260928 16:48:41.592054 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:41.592217 19582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:41.595474 19582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:41.601615 19582 catalog_manager.cc:1347] Generated new cluster ID: 7c5f0cd4b570404ab738d1b3d501d6b9
I20260928 16:48:41.601676 19582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:41.626121 19582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:41.627480 19582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:41.636668 19582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdcc51a527d448918e18ec17a25c1bd8: Generated new TSK 0
I20260928 16:48:41.637490 19582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:41.643870 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19500
W20260928 16:48:41.649705 19485 connection.cc:535] client connection to 127.0.0.1:42911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42911): Connection reset by peer (error 104)
2026-09-28T16:48:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:41 dist-test-slave-5rg1 krb5kdc[19600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:41 dist-test-slave-5rg1 krb5kdc[19600](info): set up 2 sockets
Sep 28 16:48:41 dist-test-slave-5rg1 krb5kdc[19600](info): commencing operation
krb5kdc: starting...
W20260928 16:48:42.712383 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:42 dist-test-slave-5rg1 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:42.843235 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:42.958273 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:42.958559 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:42.958602 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:42.962711 19613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:42.962782 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:42.962802 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:42.962819 19613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:42.962836 19613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:42.968210 19613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:42.969350 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:42.970320 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:42.978940 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:42.978982 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:42.978940 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:42.979202 19613 server_base.cc:1034] running on GCE node
I20260928 16:48:42.980058 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:42.981271 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:42.982467 19613 hybrid_clock.cc:648] HybridClock initialized: now 1790614122982440 us; error 41 us; skew 500 ppm
Sep 28 16:48:42 dist-test-slave-5rg1 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:42.985270 19613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:42.986209 19613 webserver.cc:460] Webserver started at http://127.0.0.1:40015/ using document root <none> and password file <none>
I20260928 16:48:42.986802 19613 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:42.986896 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:42.987133 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:42.989287 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "1275e95d7eb64179836281298addc02e"
format_stamp: "Formatted at 2026-09-28 16:48:42 on dist-test-slave-5rg1"
server_key: "a47767bd51448d13f47398e1bd9e577f"
server_key_iv: "c6a13e9558c3bab5ce64502da1d881ac"
server_key_version: "encryptionkey@0"
I20260928 16:48:42.989877 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "1275e95d7eb64179836281298addc02e"
format_stamp: "Formatted at 2026-09-28 16:48:42 on dist-test-slave-5rg1"
server_key: "a47767bd51448d13f47398e1bd9e577f"
server_key_iv: "c6a13e9558c3bab5ce64502da1d881ac"
server_key_version: "encryptionkey@0"
I20260928 16:48:42.993643 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260928 16:48:42.996224 19628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:42.997510 19613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:42.997675 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "1275e95d7eb64179836281298addc02e"
format_stamp: "Formatted at 2026-09-28 16:48:42 on dist-test-slave-5rg1"
server_key: "a47767bd51448d13f47398e1bd9e577f"
server_key_iv: "c6a13e9558c3bab5ce64502da1d881ac"
server_key_version: "encryptionkey@0"
I20260928 16:48:42.997795 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:43.002501 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:43.005580 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:43.005780 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:43.012914 19613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40401
I20260928 16:48:43.012940 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40401 every 8 connection(s)
I20260928 16:48:43.013847 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:43.016572 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:43.020056 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19613
I20260928 16:48:43.020198 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:43.020519 31504 external_mini_cluster.cc:1428] Setting key 8e5d4d977b6ea739de59b2cb97b47d55
I20260928 16:48:43.022564 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e: Bootstrap starting.
I20260928 16:48:43.024874 19681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:43.025688 19681 log.cc:824] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:43.025884 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:43.021952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46785 (local address 127.0.0.1:40401)
0928 16:48:43.022408 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:43.022423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:43.022460 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:48:43.023139 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:43.023157 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:43.023435 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:43.023818 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:43.024600 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:43.024604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:43.024616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:43.024626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:43.024676 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:43.024694 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:43.024811 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:43.024997 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:48:43.025154 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260928 16:48:43.027652 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e: No bootstrap required, opened a new log
I20260928 16:48:43.028643 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:43.031716 19681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260928 16:48:43.032015 19681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:43.032102 19681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1275e95d7eb64179836281298addc02e, State: Initialized, Role: FOLLOWER
I20260928 16:48:43.032481 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:43.030771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45759 (local address 127.0.0.1:40401)
0928 16:48:43.030885 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:43.030890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:43.031027 (+ 137us) server_negotiation.cc:407] Connection header received
0928 16:48:43.031135 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:43.031137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:43.031164 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:43.031228 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:43.032020 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:43.032026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:43.032029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:43.032043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:43.032076 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:43.032086 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:43.032194 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:43.032321 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 16:48:43.032358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:48:43.032632 19681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260928 16:48:43.032812 19681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:43.032886 19681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:43.032981 19681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:43.034003 19681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260928 16:48:43.034354 19681 leader_election.cc:302] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1275e95d7eb64179836281298addc02e; no voters:
I20260928 16:48:43.034664 19681 leader_election.cc:288] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:43.034790 19691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:43.035020 19691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [term 1 LEADER]: Becoming Leader. State: Replica: 1275e95d7eb64179836281298addc02e, State: Running, Role: LEADER
I20260928 16:48:43.035370 19691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260928 16:48:43.036006 19681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:43.037108 19693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1275e95d7eb64179836281298addc02e. Latest consensus state: current_term: 1 leader_uuid: "1275e95d7eb64179836281298addc02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260928 16:48:43.037238 19693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:43.037413 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1275e95d7eb64179836281298addc02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1275e95d7eb64179836281298addc02e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260928 16:48:43.037606 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:43.037946 19700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:43.040748 19700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:43.046394 19700 catalog_manager.cc:1347] Generated new cluster ID: 2c0bb7bf979448af84c540eb714e0707
I20260928 16:48:43.046466 19700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:43.063903 19700 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:43.064255 19700 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:43.074971 19700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1275e95d7eb64179836281298addc02e: Generated new TSK 0
I20260928 16:48:43.075793 19700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:43.085032 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19613
2026-09-28T16:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:43 dist-test-slave-5rg1 krb5kdc[19712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:43 dist-test-slave-5rg1 krb5kdc[19712](info): set up 2 sockets
Sep 28 16:48:43 dist-test-slave-5rg1 krb5kdc[19712](info): commencing operation
krb5kdc: starting...
W20260928 16:48:44.128216 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:44.259559 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:44.377949 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:44.378307 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:44.378398 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:44.382609 19725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:44.382717 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:44.382754 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:44.382802 19725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:44.382839 19725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:44.388362 19725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:44.389858 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:44.391237 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:44.400022 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:44.399959 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:44.400043 19725 server_base.cc:1034] running on GCE node
W20260928 16:48:44.399955 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:44.400725 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:44.401808 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:44.402969 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790614124402938 us; error 44 us; skew 500 ppm
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:44.414001 19725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:44.414999 19725 webserver.cc:460] Webserver started at http://127.0.0.1:38715/ using document root <none> and password file <none>
I20260928 16:48:44.415571 19725 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:44.415635 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:44.415910 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:44.418025 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "58a8f01449204f439e697908277eae97"
format_stamp: "Formatted at 2026-09-28 16:48:44 on dist-test-slave-5rg1"
server_key: "cdda9b96c3c0a797e664ff5acbf5d519"
server_key_iv: "c50b06e1cb465325d9bd9e27d996ee55"
server_key_version: "encryptionkey@0"
I20260928 16:48:44.418550 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "58a8f01449204f439e697908277eae97"
format_stamp: "Formatted at 2026-09-28 16:48:44 on dist-test-slave-5rg1"
server_key: "cdda9b96c3c0a797e664ff5acbf5d519"
server_key_iv: "c50b06e1cb465325d9bd9e27d996ee55"
server_key_version: "encryptionkey@0"
I20260928 16:48:44.422304 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:48:44.424891 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:44.426393 19725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:44.426538 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "58a8f01449204f439e697908277eae97"
format_stamp: "Formatted at 2026-09-28 16:48:44 on dist-test-slave-5rg1"
server_key: "cdda9b96c3c0a797e664ff5acbf5d519"
server_key_iv: "c50b06e1cb465325d9bd9e27d996ee55"
server_key_version: "encryptionkey@0"
I20260928 16:48:44.426653 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:44.439962 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:44.443087 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:44.443296 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:44.450783 19725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45045
I20260928 16:48:44.450803 19793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45045 every 8 connection(s)
I20260928 16:48:44.451736 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:44.454604 19794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:44.456930 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19725
I20260928 16:48:44.457087 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:44.457374 31504 external_mini_cluster.cc:1428] Setting key e7f0b1bce9ea8dbdcc4ed570e1dfff33
I20260928 16:48:44.460856 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97: Bootstrap starting.
I20260928 16:48:44.463519 19794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:44.464385 19794 log.cc:824] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:44.466545 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97: No bootstrap required, opened a new log
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:44.470389 19794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 16:48:44.470690 19794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:44.470769 19794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a8f01449204f439e697908277eae97, State: Initialized, Role: FOLLOWER
I20260928 16:48:44.471318 19794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 16:48:44.471455 19794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:44.471539 19794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:44.471663 19794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:44.472883 19794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 16:48:44.473249 19794 leader_election.cc:302] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a8f01449204f439e697908277eae97; no voters:
I20260928 16:48:44.473595 19794 leader_election.cc:288] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:44.473690 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:44.458886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51251 (local address 127.0.0.1:45045)
0928 16:48:44.459302 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:44.459317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:44.459353 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:48:44.460083 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:44.460102 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:44.460444 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:44.460725 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:44.461706 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:44.462735 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:44.463950 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:44.464423 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:44.464626 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:44.467470 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:44.467475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:44.467489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:44.467517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:44.470084 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:44.470624 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:44.470629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:44.470634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:44.470930 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:44.471230 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:44.471232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:44.471233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:44.471840 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:44.472014 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:44.472311 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 16:48:44.472658 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260928 16:48:44.473717 19799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:44.473922 19799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [term 1 LEADER]: Becoming Leader. State: Replica: 58a8f01449204f439e697908277eae97, State: Running, Role: LEADER
I20260928 16:48:44.474273 19799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260928 16:48:44.474922 19794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:44.475569 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a8f01449204f439e697908277eae97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 16:48:44.475592 19801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a8f01449204f439e697908277eae97. Latest consensus state: current_term: 1 leader_uuid: "58a8f01449204f439e697908277eae97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a8f01449204f439e697908277eae97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260928 16:48:44.475746 19801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:44.475746 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:44.476177 19806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:44.477174 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:44.479805 19806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:44.486071 19806 catalog_manager.cc:1347] Generated new cluster ID: deae553388ee4321b4a5bb406fbaae9d
I20260928 16:48:44.486191 19806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:44.490017 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:44.479488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34599 (local address 127.0.0.1:45045)
0928 16:48:44.479615 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:44.479620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:44.480636 (+ 1016us) server_negotiation.cc:407] Connection header received
0928 16:48:44.480770 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:44.480773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:44.480826 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:44.480912 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:44.481902 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:44.482800 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:44.484056 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:44.484429 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:44.484562 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:44.486312 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:44.486316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:44.486319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:44.486354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:44.488176 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:44.488712 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:44.488715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:44.488717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:44.488804 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:44.489120 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:44.489123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:44.489125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:44.489335 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:44.489457 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:44.489731 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 16:48:44.489852 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:48:44.504729 19806 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:44.506021 19806 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:44.511307 19806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a8f01449204f439e697908277eae97: Generated new TSK 0
I20260928 16:48:44.512107 19806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:44.521701 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19725
2026-09-28T16:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19825](info): set up 2 sockets
Sep 28 16:48:44 dist-test-slave-5rg1 krb5kdc[19825](info): commencing operation
krb5kdc: starting...
W20260928 16:48:45.589140 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:45 dist-test-slave-5rg1 krb5kdc[19825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:45.716252 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:45.831096 19838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:45.831387 19838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:45.831430 19838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:45.835592 19838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:45.835685 19838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:45.835705 19838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:45.835723 19838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:45.835785 19838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:45.841137 19838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:45.842300 19838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:45.843327 19838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:45.852643 19843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:45.852818 19846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:45.852584 19844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:45.852995 19838 server_base.cc:1034] running on GCE node
I20260928 16:48:45.853547 19838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:45.854734 19838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:45.855938 19838 hybrid_clock.cc:648] HybridClock initialized: now 1790614125855905 us; error 47 us; skew 500 ppm
Sep 28 16:48:45 dist-test-slave-5rg1 krb5kdc[19825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:45.859069 19838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:45.860162 19838 webserver.cc:460] Webserver started at http://127.0.0.1:38397/ using document root <none> and password file <none>
I20260928 16:48:45.860702 19838 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:45.860764 19838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:45.860942 19838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:45.863054 19838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "301169e42dda4b3a805f8715b2c36c97"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-5rg1"
server_key: "d77323ca2ed4f1bd9224f294bd011162"
server_key_iv: "03cc8e3e77e10a36de286af3e16ab3ae"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.863601 19838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "301169e42dda4b3a805f8715b2c36c97"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-5rg1"
server_key: "d77323ca2ed4f1bd9224f294bd011162"
server_key_iv: "03cc8e3e77e10a36de286af3e16ab3ae"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.867432 19838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:45.869977 19853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:45.871258 19838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:45.871368 19838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "301169e42dda4b3a805f8715b2c36c97"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-5rg1"
server_key: "d77323ca2ed4f1bd9224f294bd011162"
server_key_iv: "03cc8e3e77e10a36de286af3e16ab3ae"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.871464 19838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:45.887516 19838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:45.890666 19838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:45.890832 19838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:45.898321 19838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34269
I20260928 16:48:45.898347 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34269 every 8 connection(s)
I20260928 16:48:45.899359 19838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:45.902226 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:45.903417 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19838
I20260928 16:48:45.903565 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:45.903875 31504 external_mini_cluster.cc:1428] Setting key fd5909e004fedb97b80ed8be972b3b48
I20260928 16:48:45.908576 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97: Bootstrap starting.
I20260928 16:48:45.911343 19907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:45.912273 19907 log.cc:824] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:45.913197 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:45.905420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45693 (local address 127.0.0.1:34269)
0928 16:48:45.905828 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:45.905855 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:45.905890 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:48:45.906503 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:45.906519 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:45.906859 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:45.907133 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:45.907518 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.908565 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.909782 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.910236 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.910351 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:45.911204 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:45.911209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:45.911221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:45.911724 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:45.911782 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:45.911807 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:45.911882 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:45.912046 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:48:45.912268 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260928 16:48:45.914304 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97: No bootstrap required, opened a new log
I20260928 16:48:45.916085 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:45.918287 19907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260928 16:48:45.918617 19907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:45.918741 19907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301169e42dda4b3a805f8715b2c36c97, State: Initialized, Role: FOLLOWER
I20260928 16:48:45.919224 19907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260928 16:48:45.919386 19907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:45.919459 19907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:45.919559 19907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:45.920552 19907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260928 16:48:45.920905 19907 leader_election.cc:302] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 301169e42dda4b3a805f8715b2c36c97; no voters:
I20260928 16:48:45.921247 19907 leader_election.cc:288] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:45.921339 19917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:45.921583 19917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [term 1 LEADER]: Becoming Leader. State: Replica: 301169e42dda4b3a805f8715b2c36c97, State: Running, Role: LEADER
I20260928 16:48:45.921962 19917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260928 16:48:45.922405 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:45.918078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40807 (local address 127.0.0.1:34269)
0928 16:48:45.918214 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:45.918219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:45.918441 (+ 222us) server_negotiation.cc:407] Connection header received
0928 16:48:45.918560 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:45.918562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:45.918632 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:45.918736 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:45.919017 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.919657 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.920479 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.920775 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.920883 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:45.921779 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:45.921782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:45.921785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:45.921797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:45.921827 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:45.921840 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:45.922196 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:45.922233 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 16:48:45.922284 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:48:45.922535 19907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:45.923357 19919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 301169e42dda4b3a805f8715b2c36c97. Latest consensus state: current_term: 1 leader_uuid: "301169e42dda4b3a805f8715b2c36c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } } }
I20260928 16:48:45.923481 19919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:45.923897 19925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:45.924202 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "301169e42dda4b3a805f8715b2c36c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301169e42dda4b3a805f8715b2c36c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } } }
I20260928 16:48:45.924288 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:45.926822 19925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:45.932562 19925 catalog_manager.cc:1347] Generated new cluster ID: ec5ddf8a733d478bbec98a4a7de8ef5a
I20260928 16:48:45.932642 19925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:45.962266 19925 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:45.963303 19925 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:45.976413 19925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 301169e42dda4b3a805f8715b2c36c97: Generated new TSK 0
I20260928 16:48:45.977180 19925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:45.981063 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19838
2026-09-28T16:48:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:46 dist-test-slave-5rg1 krb5kdc[19938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:46 dist-test-slave-5rg1 krb5kdc[19938](info): set up 2 sockets
Sep 28 16:48:46 dist-test-slave-5rg1 krb5kdc[19938](info): commencing operation
krb5kdc: starting...
W20260928 16:48:47.024399 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:47 dist-test-slave-5rg1 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:47.158826 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:47.273686 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:47.274042 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:47.274128 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:47.278481 19951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:47.278604 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:47.278646 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:47.278703 19951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:47.278743 19951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:47.284590 19951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:47.285997 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:47.287164 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:47.296598 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:47.296710 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:47.296650 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:47.296612 19951 server_base.cc:1034] running on GCE node
I20260928 16:48:47.297472 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:47.298599 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:47.299778 19951 hybrid_clock.cc:648] HybridClock initialized: now 1790614127299756 us; error 34 us; skew 500 ppm
Sep 28 16:48:47 dist-test-slave-5rg1 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:47.302839 19951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:47.303984 19951 webserver.cc:460] Webserver started at http://127.0.0.1:33855/ using document root <none> and password file <none>
I20260928 16:48:47.304591 19951 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:47.304658 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:47.304939 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:47.307128 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "354eae53006b47078c05aafe17ddcea5"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-5rg1"
server_key: "9af3f6e9603239adab009bdb8c5b32cc"
server_key_iv: "8f8705fe3190c61f37f118d9b6231b2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.307829 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "354eae53006b47078c05aafe17ddcea5"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-5rg1"
server_key: "9af3f6e9603239adab009bdb8c5b32cc"
server_key_iv: "8f8705fe3190c61f37f118d9b6231b2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.311939 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:48:47.314571 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:47.316280 19951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:47.316418 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "354eae53006b47078c05aafe17ddcea5"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-5rg1"
server_key: "9af3f6e9603239adab009bdb8c5b32cc"
server_key_iv: "8f8705fe3190c61f37f118d9b6231b2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.316571 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:47.321448 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:47.324914 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:47.325124 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:47.332823 19951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40287
I20260928 16:48:47.332854 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40287 every 8 connection(s)
I20260928 16:48:47.333802 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:47.335973 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 19951
I20260928 16:48:47.336103 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:47.336419 31504 external_mini_cluster.cc:1428] Setting key b0d9dcc34a181387812ab1f1a67118e6
I20260928 16:48:47.336825 20019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:47.341552 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:47.337810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36329 (local address 127.0.0.1:40287)
0928 16:48:47.338211 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:47.338229 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:47.338272 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:48:47.338912 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:47.338928 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:47.339188 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:47.339461 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:47.340298 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:47.340303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:47.340314 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:47.340323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:47.340382 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:47.340400 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:47.340507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:47.340619 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 16:48:47.340775 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260928 16:48:47.343858 20019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5: Bootstrap starting.
I20260928 16:48:47.344468 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:47.346184 20019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:47.347054 20019 log.cc:824] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:47.348537 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:47.346624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52043 (local address 127.0.0.1:40287)
0928 16:48:47.346745 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:47.346748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:47.346974 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:48:47.347127 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:47.347130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:47.347191 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:47.347264 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:47.348095 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:47.348099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:47.348101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:47.348113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:47.348143 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:47.348160 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:47.348250 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:47.348325 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 16:48:47.348410 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:48:47.349231 20019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5: No bootstrap required, opened a new log
I20260928 16:48:47.353046 20019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } }
I20260928 16:48:47.353307 20019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:47.353390 20019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354eae53006b47078c05aafe17ddcea5, State: Initialized, Role: FOLLOWER
I20260928 16:48:47.353893 20019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } }
I20260928 16:48:47.354053 20019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:47.354143 20019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:47.354243 20019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:47.355201 20019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } }
I20260928 16:48:47.355561 20019 leader_election.cc:302] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 354eae53006b47078c05aafe17ddcea5; no voters:
I20260928 16:48:47.355926 20019 leader_election.cc:288] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:47.356117 20029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:47.356436 20029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [term 1 LEADER]: Becoming Leader. State: Replica: 354eae53006b47078c05aafe17ddcea5, State: Running, Role: LEADER
I20260928 16:48:47.356781 20029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } }
I20260928 16:48:47.357287 20019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:47.358608 20030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "354eae53006b47078c05aafe17ddcea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } } }
I20260928 16:48:47.358758 20030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:47.359067 20031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354eae53006b47078c05aafe17ddcea5. Latest consensus state: current_term: 1 leader_uuid: "354eae53006b47078c05aafe17ddcea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354eae53006b47078c05aafe17ddcea5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40287 } } }
I20260928 16:48:47.359166 20031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:47.359778 20040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:47.362778 20040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:47.368477 20040 catalog_manager.cc:1347] Generated new cluster ID: 623983c3756243f8b7c43b09ae6f6128
I20260928 16:48:47.368564 20040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:47.383178 20040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:47.383548 20040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:47.390419 20040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 354eae53006b47078c05aafe17ddcea5: Generated new TSK 0
I20260928 16:48:47.391145 20040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:47.396487 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 19951
2026-09-28T16:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:47 dist-test-slave-5rg1 krb5kdc[20050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:47 dist-test-slave-5rg1 krb5kdc[20050](info): set up 2 sockets
Sep 28 16:48:47 dist-test-slave-5rg1 krb5kdc[20050](info): commencing operation
krb5kdc: starting...
W20260928 16:48:48.440244 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:48.572154 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:48.688824 20064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:48.689239 20064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:48.689332 20064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:48.693581 20064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:48.693679 20064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:48.693719 20064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:48.693755 20064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:48.693795 20064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:48.699483 20064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:48.700659 20064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:48.701715 20064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:48.711088 20072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:48.711063 20064 server_base.cc:1034] running on GCE node
W20260928 16:48:48.711020 20069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:48.711020 20070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:48.711920 20064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:48.713039 20064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:48.714205 20064 hybrid_clock.cc:648] HybridClock initialized: now 1790614128714194 us; error 25 us; skew 500 ppm
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:48.716984 20064 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:48.717881 20064 webserver.cc:460] Webserver started at http://127.0.0.1:40877/ using document root <none> and password file <none>
I20260928 16:48:48.718458 20064 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:48.718523 20064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:48.718751 20064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:48.720891 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "4cbcce9e1d3d4704aeb3805a1291de99"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-5rg1"
server_key: "0c59c1d5b1a7df978dbe040df36729ce"
server_key_iv: "46177f4493bd4d362daf8abb05c68dcf"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.721467 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "4cbcce9e1d3d4704aeb3805a1291de99"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-5rg1"
server_key: "0c59c1d5b1a7df978dbe040df36729ce"
server_key_iv: "46177f4493bd4d362daf8abb05c68dcf"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.725215 20064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:48:48.727651 20079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:48.728962 20064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:48:48.729110 20064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "4cbcce9e1d3d4704aeb3805a1291de99"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-5rg1"
server_key: "0c59c1d5b1a7df978dbe040df36729ce"
server_key_iv: "46177f4493bd4d362daf8abb05c68dcf"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.729230 20064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:48.742313 20064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:48.745253 20064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:48.745462 20064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:48.753002 20064 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44207
I20260928 16:48:48.753026 20132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44207 every 8 connection(s)
I20260928 16:48:48.753995 20064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:48.756780 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:48.759157 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20064
I20260928 16:48:48.759265 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:48.759550 31504 external_mini_cluster.cc:1428] Setting key 2673ebff9b8df5bda7942e27d94d03e4
I20260928 16:48:48.763299 20133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99: Bootstrap starting.
I20260928 16:48:48.765975 20133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:48.766894 20133 log.cc:824] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:48.769132 20133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99: No bootstrap required, opened a new log
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:48.774139 20133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } }
I20260928 16:48:48.774456 20133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:48.774519 20133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbcce9e1d3d4704aeb3805a1291de99, State: Initialized, Role: FOLLOWER
I20260928 16:48:48.775020 20133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } }
I20260928 16:48:48.775178 20133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:48.775244 20133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:48.775339 20133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:48.775425 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:48.760978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:41287 (local address 127.0.0.1:44207)
0928 16:48:48.761410 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:48.761433 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:48.761477 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:48:48.762302 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:48.762327 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:48.762766 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:48.763120 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:48.763932 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:48.765414 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:48.766318 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:48.767024 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:48.767179 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:48.769841 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:48.769847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:48.769865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:48.769899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:48.772025 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:48.772550 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:48.772556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:48.772563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:48.772705 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:48.773030 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:48.773034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:48.773036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:48.773621 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:48.773854 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:48.774145 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 16:48:48.774383 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260928 16:48:48.776618 20133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } }
I20260928 16:48:48.776990 20133 leader_election.cc:302] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbcce9e1d3d4704aeb3805a1291de99; no voters:
I20260928 16:48:48.777398 20133 leader_election.cc:288] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:48.777529 20138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:48.777796 20138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [term 1 LEADER]: Becoming Leader. State: Replica: 4cbcce9e1d3d4704aeb3805a1291de99, State: Running, Role: LEADER
I20260928 16:48:48.778196 20138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } }
I20260928 16:48:48.778784 20133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:48.779016 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:48.780097 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbcce9e1d3d4704aeb3805a1291de99. Latest consensus state: current_term: 1 leader_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } } }
I20260928 16:48:48.780210 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:48.780601 20147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:48.781435 20140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbcce9e1d3d4704aeb3805a1291de99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44207 } } }
I20260928 16:48:48.781549 20140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:48.784487 20147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:48.791538 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:48.782092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:54873 (local address 127.0.0.1:44207)
0928 16:48:48.782208 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:48.782213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:48.782513 (+ 300us) server_negotiation.cc:407] Connection header received
0928 16:48:48.782680 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:48.782684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:48.782734 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:48.782813 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:48.783731 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:48.784446 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:48.785759 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:48.786097 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:48.786207 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:48.788051 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:48.788054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:48.788057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:48.788084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:48.789746 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:48.790291 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:48.790294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:48.790295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:48.790368 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:48.790711 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:48.790713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:48.790714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:48.790879 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:48.790986 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:48.791282 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 16:48:48.791363 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:48:48.793202 20147 catalog_manager.cc:1347] Generated new cluster ID: b1c3ba14c9994cebabe23c7e558ae051
I20260928 16:48:48.793267 20147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:48.798416 20147 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:48.799729 20147 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:48.807384 20147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cbcce9e1d3d4704aeb3805a1291de99: Generated new TSK 0
I20260928 16:48:48.808279 20147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:48.818159 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20064
W20260928 16:48:48.824180 20048 connection.cc:535] client connection to 127.0.0.1:44207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44207): Connection reset by peer (error 104)
2026-09-28T16:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20165](info): set up 2 sockets
Sep 28 16:48:48 dist-test-slave-5rg1 krb5kdc[20165](info): commencing operation
krb5kdc: starting...
W20260928 16:48:49.864367 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:49 dist-test-slave-5rg1 krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:49.991875 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:50.104655 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:50.104949 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:50.104992 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:50.109282 20178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:50.109352 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:50.109372 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:50.109390 20178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:50.109436 20178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:50.114984 20178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:50.116293 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:50.117614 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:50.127236 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:50.127228 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:50.127228 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:50.127578 20178 server_base.cc:1034] running on GCE node
I20260928 16:48:50.128207 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:50.129379 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:50.130575 20178 hybrid_clock.cc:648] HybridClock initialized: now 1790614130130548 us; error 41 us; skew 500 ppm
Sep 28 16:48:50 dist-test-slave-5rg1 krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:50.138661 20178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:50.139838 20178 webserver.cc:460] Webserver started at http://127.0.0.1:33131/ using document root <none> and password file <none>
I20260928 16:48:50.140448 20178 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:50.140550 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:50.140805 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:50.142968 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "5e849316cb4146ddb6579e0be3f4e7d1"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-5rg1"
server_key: "32781a16a4160efe915f54dd24970d40"
server_key_iv: "12aa61c3935001f461601c81aec42c47"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.143561 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "5e849316cb4146ddb6579e0be3f4e7d1"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-5rg1"
server_key: "32781a16a4160efe915f54dd24970d40"
server_key_iv: "12aa61c3935001f461601c81aec42c47"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.147360 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:50.149967 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:50.151244 20178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:50.151404 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "5e849316cb4146ddb6579e0be3f4e7d1"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-5rg1"
server_key: "32781a16a4160efe915f54dd24970d40"
server_key_iv: "12aa61c3935001f461601c81aec42c47"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.151530 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:50.172091 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:50.175092 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:50.175300 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:50.182581 20178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44705
I20260928 16:48:50.182600 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44705 every 8 connection(s)
I20260928 16:48:50.183707 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:50.186547 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:50.189044 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20178
I20260928 16:48:50.189198 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:50.189517 31504 external_mini_cluster.cc:1428] Setting key 1852303c8e3c24d4bb757ef70ebd276a
I20260928 16:48:50.193141 20247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1: Bootstrap starting.
I20260928 16:48:50.195371 20247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:50.196197 20247 log.cc:824] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:50.198583 20247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1: No bootstrap required, opened a new log
I20260928 16:48:50.199267 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:50.191023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:46699 (local address 127.0.0.1:44705)
0928 16:48:50.191506 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:50.191521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:50.191558 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:48:50.192356 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:50.192380 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:50.192729 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:50.193094 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:50.193501 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.194486 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.195421 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.196032 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.196215 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:50.197169 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:50.197179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:50.197197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:50.197701 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:50.197771 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:50.197803 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:50.197942 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:50.198170 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:48:50.198297 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":139,"threads_started":1}
I20260928 16:48:50.202281 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:50.202865 20247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260928 16:48:50.203085 20247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:50.203145 20247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e849316cb4146ddb6579e0be3f4e7d1, State: Initialized, Role: FOLLOWER
I20260928 16:48:50.203577 20247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260928 16:48:50.203814 20247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:50.203903 20247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:50.204003 20247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:50.205228 20247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260928 16:48:50.205614 20247 leader_election.cc:302] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e849316cb4146ddb6579e0be3f4e7d1; no voters:
I20260928 16:48:50.205950 20247 leader_election.cc:288] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:50.206041 20257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:50.206307 20257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 5e849316cb4146ddb6579e0be3f4e7d1, State: Running, Role: LEADER
I20260928 16:48:50.206701 20257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } }
I20260928 16:48:50.207290 20247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:50.208458 20258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } } }
I20260928 16:48:50.208673 20258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:50.208918 20257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e849316cb4146ddb6579e0be3f4e7d1. Latest consensus state: current_term: 1 leader_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e849316cb4146ddb6579e0be3f4e7d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44705 } } }
I20260928 16:48:50.209071 20257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:50.209527 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:50.204539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:52917 (local address 127.0.0.1:44705)
0928 16:48:50.204659 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:50.204664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:50.204820 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:48:50.204902 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:50.204905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:50.204952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:50.205042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:50.205301 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.206065 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.207185 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.207517 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.207639 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:50.208782 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:50.208786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:50.208789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:50.208819 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:50.208851 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:50.208869 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:50.208970 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:50.209360 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 16:48:50.209397 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:48:50.210672 20266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:50.213397 20266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:50.219105 20266 catalog_manager.cc:1347] Generated new cluster ID: 533dcef6531d49eea0c1822406716f3d
I20260928 16:48:50.219173 20266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:50.230129 20266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:50.231065 20266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:50.235606 20266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e849316cb4146ddb6579e0be3f4e7d1: Generated new TSK 0
I20260928 16:48:50.236357 20266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:50.239987 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20178
W20260928 16:48:50.246629 20163 connection.cc:535] client connection to 127.0.0.1:44705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44705): Connection reset by peer (error 104)
2026-09-28T16:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:50 dist-test-slave-5rg1 krb5kdc[20278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:50 dist-test-slave-5rg1 krb5kdc[20278](info): set up 2 sockets
Sep 28 16:48:50 dist-test-slave-5rg1 krb5kdc[20278](info): commencing operation
krb5kdc: starting...
W20260928 16:48:51.300203 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:51 dist-test-slave-5rg1 krb5kdc[20278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:51.427184 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:51.539718 20291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:51.540030 20291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:51.540102 20291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:51.544180 20291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:51.544269 20291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:51.544306 20291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:51.544366 20291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:51.544404 20291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:51.549763 20291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:51.550874 20291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:51.551950 20291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:51.560592 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:51.560592 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:51.560642 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:51.560846 20291 server_base.cc:1034] running on GCE node
I20260928 16:48:51.561565 20291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:51.562800 20291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:51.564008 20291 hybrid_clock.cc:648] HybridClock initialized: now 1790614131563988 us; error 30 us; skew 500 ppm
Sep 28 16:48:51 dist-test-slave-5rg1 krb5kdc[20278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:51.566712 20291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:51.567657 20291 webserver.cc:460] Webserver started at http://127.0.0.1:42397/ using document root <none> and password file <none>
I20260928 16:48:51.568235 20291 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:51.568293 20291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:51.568462 20291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:51.570458 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "d400e10a9f9d4da3a07e2da4a100faa3"
format_stamp: "Formatted at 2026-09-28 16:48:51 on dist-test-slave-5rg1"
server_key: "5085f222629b65c2d07a5f742cdd2686"
server_key_iv: "ec68253250fd4fbaa24f0ac8b0426806"
server_key_version: "encryptionkey@0"
I20260928 16:48:51.570981 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "d400e10a9f9d4da3a07e2da4a100faa3"
format_stamp: "Formatted at 2026-09-28 16:48:51 on dist-test-slave-5rg1"
server_key: "5085f222629b65c2d07a5f742cdd2686"
server_key_iv: "ec68253250fd4fbaa24f0ac8b0426806"
server_key_version: "encryptionkey@0"
I20260928 16:48:51.574649 20291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:48:51.577046 20306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:51.578325 20291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:51.578464 20291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "d400e10a9f9d4da3a07e2da4a100faa3"
format_stamp: "Formatted at 2026-09-28 16:48:51 on dist-test-slave-5rg1"
server_key: "5085f222629b65c2d07a5f742cdd2686"
server_key_iv: "ec68253250fd4fbaa24f0ac8b0426806"
server_key_version: "encryptionkey@0"
I20260928 16:48:51.578589 20291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:51.583247 20291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:51.586422 20291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:51.586632 20291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:51.594241 20291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35575
I20260928 16:48:51.594267 20358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35575 every 8 connection(s)
I20260928 16:48:51.595218 20291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:51.598104 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:51.603772 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: Bootstrap starting.
I20260928 16:48:51.603796 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20291
I20260928 16:48:51.603948 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:51.604260 31504 external_mini_cluster.cc:1428] Setting key 7aafd80848b14fe8fa50755e06f70cac
I20260928 16:48:51.606459 20359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:51.607205 20359 log.cc:824] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:51.609426 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: No bootstrap required, opened a new log
I20260928 16:48:51.609688 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:51.605668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:60843 (local address 127.0.0.1:35575)
0928 16:48:51.606122 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:51.606137 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:51.606171 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:48:51.606870 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:51.606889 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:51.607182 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:51.607554 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:51.608418 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:51.608422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:51.608436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:51.608447 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:51.608501 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:51.608521 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:51.608642 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:51.608755 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 16:48:51.608935 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260928 16:48:51.612627 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:51.613686 20359 raft_consensus.cc:348] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260928 16:48:51.613965 20359 raft_consensus.cc:374] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:51.614025 20359 raft_consensus.cc:729] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d400e10a9f9d4da3a07e2da4a100faa3, State: Initialized, Role: FOLLOWER
I20260928 16:48:51.614442 20359 consensus_queue.cc:260] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260928 16:48:51.614593 20359 raft_consensus.cc:388] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:51.614660 20359 raft_consensus.cc:482] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:51.614818 20359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:51.615823 20359 raft_consensus.cc:504] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260928 16:48:51.616174 20359 leader_election.cc:302] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d400e10a9f9d4da3a07e2da4a100faa3; no voters:
I20260928 16:48:51.616504 20359 leader_election.cc:288] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:51.616583 20369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:51.616690 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:51.614792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:58049 (local address 127.0.0.1:35575)
0928 16:48:51.614922 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:51.614927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:51.615165 (+ 238us) server_negotiation.cc:407] Connection header received
0928 16:48:51.615305 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:51.615308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:51.615349 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:51.615435 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:51.616195 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:51.616198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:51.616201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:51.616213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:51.616240 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:51.616259 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:51.616358 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:51.616455 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 16:48:51.616503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:48:51.617892 20359 sys_catalog.cc:564] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:51.618613 20369 raft_consensus.cc:686] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [term 1 LEADER]: Becoming Leader. State: Replica: d400e10a9f9d4da3a07e2da4a100faa3, State: Running, Role: LEADER
I20260928 16:48:51.618948 20369 consensus_queue.cc:237] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260928 16:48:51.620673 20371 sys_catalog.cc:455] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d400e10a9f9d4da3a07e2da4a100faa3. Latest consensus state: current_term: 1 leader_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } } }
I20260928 16:48:51.621070 20371 sys_catalog.cc:458] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:51.620874 20372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d400e10a9f9d4da3a07e2da4a100faa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } } }
I20260928 16:48:51.621366 20372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3 [sys.catalog]: This master's current role is: LEADER
W20260928 16:48:51.624395 20384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:48:51.624500 20384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:48:51.624580 20385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:51.625341 20385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:51.630796 20385 catalog_manager.cc:1347] Generated new cluster ID: 5744946437cf408caa5624b45472608c
I20260928 16:48:51.630882 20385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:51.656724 20385 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:51.657086 20385 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:51.674930 20385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d400e10a9f9d4da3a07e2da4a100faa3: Generated new TSK 0
I20260928 16:48:51.675872 20385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:51.686187 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20291
2026-09-28T16:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:51 dist-test-slave-5rg1 krb5kdc[20391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:51 dist-test-slave-5rg1 krb5kdc[20391](info): set up 2 sockets
Sep 28 16:48:51 dist-test-slave-5rg1 krb5kdc[20391](info): commencing operation
krb5kdc: starting...
W20260928 16:48:52.736141 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:52 dist-test-slave-5rg1 krb5kdc[20391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:52.863714 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:52.975188 20404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:52.975487 20404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:52.975533 20404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:52.979758 20404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:52.979830 20404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:52.979849 20404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:52.979866 20404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:52.979882 20404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:52.985471 20404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:52.986598 20404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:52.987619 20404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:52.996845 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:52.996847 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:52.996881 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:52.997351 20404 server_base.cc:1034] running on GCE node
I20260928 16:48:52.997941 20404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:52.999071 20404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:53.000267 20404 hybrid_clock.cc:648] HybridClock initialized: now 1790614133000239 us; error 45 us; skew 500 ppm
Sep 28 16:48:52 dist-test-slave-5rg1 krb5kdc[20391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:53.003096 20404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:53.004122 20404 webserver.cc:460] Webserver started at http://127.0.0.1:37021/ using document root <none> and password file <none>
I20260928 16:48:53.004693 20404 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:53.004781 20404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:53.005016 20404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:53.007138 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "c80337dd6dff472db2b37d3fd0d29bba"
format_stamp: "Formatted at 2026-09-28 16:48:53 on dist-test-slave-5rg1"
server_key: "32fc87a5341009c6b626558e00cd7294"
server_key_iv: "c646197d1ac7e08f1d24fa575464a271"
server_key_version: "encryptionkey@0"
I20260928 16:48:53.007807 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "c80337dd6dff472db2b37d3fd0d29bba"
format_stamp: "Formatted at 2026-09-28 16:48:53 on dist-test-slave-5rg1"
server_key: "32fc87a5341009c6b626558e00cd7294"
server_key_iv: "c646197d1ac7e08f1d24fa575464a271"
server_key_version: "encryptionkey@0"
I20260928 16:48:53.011744 20404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:53.014315 20419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:53.015616 20404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:53.015790 20404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "c80337dd6dff472db2b37d3fd0d29bba"
format_stamp: "Formatted at 2026-09-28 16:48:53 on dist-test-slave-5rg1"
server_key: "32fc87a5341009c6b626558e00cd7294"
server_key_iv: "c646197d1ac7e08f1d24fa575464a271"
server_key_version: "encryptionkey@0"
I20260928 16:48:53.015921 20404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:53.040827 20404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:53.043881 20404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:53.044116 20404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:53.051383 20404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33623
I20260928 16:48:53.051412 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33623 every 8 connection(s)
I20260928 16:48:53.052341 20404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:53.055074 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:53.060504 20472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba: Bootstrap starting.
I20260928 16:48:53.060681 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20404
I20260928 16:48:53.060827 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:53.061122 31504 external_mini_cluster.cc:1428] Setting key 18d6ad8f1e3a23ec9c0c7fa42ae758be
I20260928 16:48:53.063151 20472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:53.063961 20472 log.cc:824] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:53.066174 20472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba: No bootstrap required, opened a new log
I20260928 16:48:53.071388 20472 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260928 16:48:53.071733 20472 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:53.071815 20472 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80337dd6dff472db2b37d3fd0d29bba, State: Initialized, Role: FOLLOWER
Sep 28 16:48:53 dist-test-slave-5rg1 krb5kdc[20391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:53.072389 20472 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260928 16:48:53.072602 20472 raft_consensus.cc:388] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:53.072670 20472 raft_consensus.cc:482] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:53.072769 20472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:53.074018 20472 raft_consensus.cc:504] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260928 16:48:53.074409 20472 leader_election.cc:302] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c80337dd6dff472db2b37d3fd0d29bba; no voters:
I20260928 16:48:53.074748 20472 leader_election.cc:288] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:53.074878 20477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:53.075116 20477 raft_consensus.cc:686] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [term 1 LEADER]: Becoming Leader. State: Replica: c80337dd6dff472db2b37d3fd0d29bba, State: Running, Role: LEADER
I20260928 16:48:53.075515 20477 consensus_queue.cc:237] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } }
I20260928 16:48:53.076257 20472 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:53.076833 20479 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c80337dd6dff472db2b37d3fd0d29bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } } }
I20260928 16:48:53.077039 20479 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:53.077275 20478 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c80337dd6dff472db2b37d3fd0d29bba. Latest consensus state: current_term: 1 leader_uuid: "c80337dd6dff472db2b37d3fd0d29bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80337dd6dff472db2b37d3fd0d29bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33623 } } }
I20260928 16:48:53.077378 20478 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:53.077572 20486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:53.079351 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:53.062452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:56881 (local address 127.0.0.1:33623)
0928 16:48:53.062968 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:53.062985 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:53.063031 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:48:53.063872 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:53.063893 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:53.064320 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:53.064722 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:53.065661 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:53.067171 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:53.068556 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:53.069121 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:53.069279 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:53.072964 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:53.072970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:53.072985 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:53.073019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:53.075109 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:53.075748 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:53.075754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:53.075760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:53.075896 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:53.076317 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:53.076322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:53.076324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:53.076843 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:53.077075 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:53.077914 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 16:48:53.078204 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":133,"threads_started":1}
I20260928 16:48:53.081141 20486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:53.082886 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:53.087334 20486 catalog_manager.cc:1347] Generated new cluster ID: e61348327c624d9795f16d46b57f6232
I20260928 16:48:53.087409 20486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:53.094717 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:53.085192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:49987 (local address 127.0.0.1:33623)
0928 16:48:53.085332 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:53.085338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:53.085781 (+ 443us) server_negotiation.cc:407] Connection header received
0928 16:48:53.085941 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:53.085944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:53.085997 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:53.086091 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:53.086833 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:53.087592 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:53.088829 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:53.089167 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:53.089261 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:53.091135 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:53.091139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:53.091142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:53.091174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:53.092851 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:53.093408 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:53.093411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:53.093414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:53.093501 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:53.093829 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:53.093831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:53.093833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:53.094036 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:53.094167 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:53.094459 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 16:48:53.094554 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 16:48:53.103730 20486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:53.104761 20486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:53.111012 20486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c80337dd6dff472db2b37d3fd0d29bba: Generated new TSK 0
I20260928 16:48:53.111935 20486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:53.117127 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20404
W20260928 16:48:53.123005 20389 connection.cc:535] client connection to 127.0.0.1:33623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33623): Connection reset by peer (error 104)
2026-09-28T16:48:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:53 dist-test-slave-5rg1 krb5kdc[20503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:53 dist-test-slave-5rg1 krb5kdc[20503](info): set up 2 sockets
Sep 28 16:48:53 dist-test-slave-5rg1 krb5kdc[20503](info): commencing operation
krb5kdc: starting...
W20260928 16:48:54.160094 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:54 dist-test-slave-5rg1 krb5kdc[20503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:54.287637 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:54.400667 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:54.400998 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:54.401101 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:54.405267 20517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:54.405372 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:54.405416 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:54.405454 20517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:54.405483 20517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:54.410971 20517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:54.412137 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:54.413187 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:54.422075 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:54.422041 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:54.422327 20517 server_base.cc:1034] running on GCE node
W20260928 16:48:54.422063 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:54.423061 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:54.424229 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:54.425417 20517 hybrid_clock.cc:648] HybridClock initialized: now 1790614134425384 us; error 51 us; skew 500 ppm
Sep 28 16:48:54 dist-test-slave-5rg1 krb5kdc[20503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:54.428128 20517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:54.429045 20517 webserver.cc:460] Webserver started at http://127.0.0.1:36179/ using document root <none> and password file <none>
I20260928 16:48:54.429613 20517 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:54.429708 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:54.429939 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:54.432067 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "3fb876625e944ce48407b86cb403b673"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-5rg1"
server_key: "9a3720a9e864776f359c0bfead685f4e"
server_key_iv: "92fd2cc2ff76826ec74a5b51106572d8"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.432624 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "3fb876625e944ce48407b86cb403b673"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-5rg1"
server_key: "9a3720a9e864776f359c0bfead685f4e"
server_key_iv: "92fd2cc2ff76826ec74a5b51106572d8"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.436548 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:48:54.439124 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:54.440485 20517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:54.440620 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "3fb876625e944ce48407b86cb403b673"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-5rg1"
server_key: "9a3720a9e864776f359c0bfead685f4e"
server_key_iv: "92fd2cc2ff76826ec74a5b51106572d8"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.440733 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:54.460628 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:54.463838 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:54.464080 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:54.471859 20517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36751
I20260928 16:48:54.471894 20584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36751 every 8 connection(s)
I20260928 16:48:54.472853 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:54.474660 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20517
I20260928 16:48:54.474808 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:54.475129 31504 external_mini_cluster.cc:1428] Setting key b01d0a83c24e5d451fb621d487427564
I20260928 16:48:54.476332 20585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:54.482705 20585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673: Bootstrap starting.
I20260928 16:48:54.484301 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:54.476579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:49667 (local address 127.0.0.1:36751)
0928 16:48:54.476981 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:54.477043 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:54.477078 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:48:54.477704 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:54.477719 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:54.478056 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:54.478330 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:54.478709 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:54.479778 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:54.480981 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:54.481430 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:54.481542 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:54.482493 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:54.482497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:54.482510 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:54.482942 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:54.483000 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:54.483023 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:54.483141 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:54.483370 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 16:48:54.483514 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260928 16:48:54.485276 20585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:54.486055 20585 log.cc:824] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:54.487320 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:54.488250 20585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673: No bootstrap required, opened a new log
I20260928 16:48:54.492796 20585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260928 16:48:54.493122 20585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:54.493192 20585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb876625e944ce48407b86cb403b673, State: Initialized, Role: FOLLOWER
I20260928 16:48:54.493515 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:54.489442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:35831 (local address 127.0.0.1:36751)
0928 16:48:54.489558 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:54.489563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:54.489708 (+ 145us) server_negotiation.cc:407] Connection header received
0928 16:48:54.489815 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:54.489817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:54.489853 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:54.489917 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:54.490240 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:54.490735 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:54.491879 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:54.492088 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:54.492165 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:54.492987 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:54.492990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:54.492992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:54.493003 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:54.493045 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:54.493055 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:54.493164 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:54.493297 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:48:54.493336 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:48:54.493848 20585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260928 16:48:54.494029 20585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:54.494098 20585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:54.494200 20585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:54.495446 20585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260928 16:48:54.495921 20585 leader_election.cc:302] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fb876625e944ce48407b86cb403b673; no voters:
I20260928 16:48:54.496309 20585 leader_election.cc:288] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:54.496400 20595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:54.496665 20595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb876625e944ce48407b86cb403b673, State: Running, Role: LEADER
I20260928 16:48:54.496971 20595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260928 16:48:54.497570 20585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:54.498582 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fb876625e944ce48407b86cb403b673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260928 16:48:54.498747 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:54.499114 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fb876625e944ce48407b86cb403b673. Latest consensus state: current_term: 1 leader_uuid: "3fb876625e944ce48407b86cb403b673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb876625e944ce48407b86cb403b673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260928 16:48:54.499264 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:54.499887 20604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:54.502858 20604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:54.508103 20604 catalog_manager.cc:1347] Generated new cluster ID: 159dbb98fa8d4749b6f3b19d02947128
I20260928 16:48:54.508175 20604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:54.524518 20604 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:54.525542 20604 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:54.533630 20604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fb876625e944ce48407b86cb403b673: Generated new TSK 0
I20260928 16:48:54.534336 20604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:54.540100 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20517
W20260928 16:48:54.546291 20501 connection.cc:535] client connection to 127.0.0.1:36751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36751): Connection reset by peer (error 104)
2026-09-28T16:48:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:54 dist-test-slave-5rg1 krb5kdc[20616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:54 dist-test-slave-5rg1 krb5kdc[20616](info): set up 2 sockets
Sep 28 16:48:54 dist-test-slave-5rg1 krb5kdc[20616](info): commencing operation
krb5kdc: starting...
W20260928 16:48:55.608340 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:55 dist-test-slave-5rg1 krb5kdc[20616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:55.735785 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:55.848429 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:55.848696 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:55.848737 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:55.852742 20629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:55.852799 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:55.852818 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:55.852834 20629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:55.852849 20629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:55.858047 20629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:55.859066 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:55.860109 20629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:55.869410 20635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:55.869400 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:55.869450 20629 server_base.cc:1034] running on GCE node
W20260928 16:48:55.869400 20634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:55.870232 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:55.871464 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:55.872678 20629 hybrid_clock.cc:648] HybridClock initialized: now 1790614135872647 us; error 45 us; skew 500 ppm
Sep 28 16:48:55 dist-test-slave-5rg1 krb5kdc[20616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:55.886266 20629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:55.887477 20629 webserver.cc:460] Webserver started at http://127.0.0.1:46435/ using document root <none> and password file <none>
I20260928 16:48:55.888190 20629 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:55.888296 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:55.888566 20629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:55.890888 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "6e8a775128104cb9af9bd8264ad5bc7b"
format_stamp: "Formatted at 2026-09-28 16:48:55 on dist-test-slave-5rg1"
server_key: "4c39e6224e990a215bd4d7d731dc4201"
server_key_iv: "7565823d616be908821df25c26e3c794"
server_key_version: "encryptionkey@0"
I20260928 16:48:55.891542 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "6e8a775128104cb9af9bd8264ad5bc7b"
format_stamp: "Formatted at 2026-09-28 16:48:55 on dist-test-slave-5rg1"
server_key: "4c39e6224e990a215bd4d7d731dc4201"
server_key_iv: "7565823d616be908821df25c26e3c794"
server_key_version: "encryptionkey@0"
I20260928 16:48:55.895625 20629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:48:55.898643 20645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:55.900173 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:55.900384 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "6e8a775128104cb9af9bd8264ad5bc7b"
format_stamp: "Formatted at 2026-09-28 16:48:55 on dist-test-slave-5rg1"
server_key: "4c39e6224e990a215bd4d7d731dc4201"
server_key_iv: "7565823d616be908821df25c26e3c794"
server_key_version: "encryptionkey@0"
I20260928 16:48:55.900568 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:55.906327 20629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:55.910059 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:55.910400 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:55.918434 20629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35921
I20260928 16:48:55.918434 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35921 every 8 connection(s)
I20260928 16:48:55.919535 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:55.922472 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20629
I20260928 16:48:55.922596 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:55.922633 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:55.922942 31504 external_mini_cluster.cc:1428] Setting key 6613cc0864b3200b71fefdfd1bf6682b
I20260928 16:48:55.928970 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:55.924502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:43769 (local address 127.0.0.1:35921)
0928 16:48:55.924953 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:55.924969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:55.925006 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:48:55.926025 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:55.926045 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:55.926352 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:55.926724 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:55.927605 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:55.927610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:55.927622 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:55.927635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:55.927754 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:55.927780 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:55.927957 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:55.928034 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:48:55.928207 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260928 16:48:55.929589 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b: Bootstrap starting.
I20260928 16:48:55.932004 20699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:55.932057 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:55.932757 20699 log.cc:824] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:55.934998 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b: No bootstrap required, opened a new log
I20260928 16:48:55.936230 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:55.934385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.223:35905 (local address 127.0.0.1:35921)
0928 16:48:55.934534 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:55.934538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:55.934694 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:48:55.934800 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:55.934803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:55.934838 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:55.934928 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:55.935773 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:55.935776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:55.935779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:55.935791 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:55.935822 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:55.935835 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:55.935927 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:55.936069 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:48:55.936112 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:55.939183 20699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } }
I20260928 16:48:55.939419 20699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:55.939459 20699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e8a775128104cb9af9bd8264ad5bc7b, State: Initialized, Role: FOLLOWER
I20260928 16:48:55.939975 20699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } }
I20260928 16:48:55.940104 20699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:55.940186 20699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:55.940284 20699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:55.941272 20699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } }
I20260928 16:48:55.941637 20699 leader_election.cc:302] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e8a775128104cb9af9bd8264ad5bc7b; no voters:
I20260928 16:48:55.941960 20699 leader_election.cc:288] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:55.942127 20709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:55.942384 20709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [term 1 LEADER]: Becoming Leader. State: Replica: 6e8a775128104cb9af9bd8264ad5bc7b, State: Running, Role: LEADER
I20260928 16:48:55.942701 20709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } }
I20260928 16:48:55.943401 20699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:55.944742 20711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e8a775128104cb9af9bd8264ad5bc7b. Latest consensus state: current_term: 1 leader_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } } }
I20260928 16:48:55.944764 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e8a775128104cb9af9bd8264ad5bc7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35921 } } }
I20260928 16:48:55.944887 20711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:55.944886 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:55.945426 20719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:55.949155 20719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:55.955338 20719 catalog_manager.cc:1347] Generated new cluster ID: 9dadbb7b1a0b4332b01bb9348280a811
I20260928 16:48:55.955433 20719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:55.970846 20719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:55.971185 20719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:55.977020 20719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e8a775128104cb9af9bd8264ad5bc7b: Generated new TSK 0
I20260928 16:48:55.977898 20719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:55.984354 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20629
2026-09-28T16:48:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:56 dist-test-slave-5rg1 krb5kdc[20730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:56 dist-test-slave-5rg1 krb5kdc[20730](info): set up 2 sockets
Sep 28 16:48:56 dist-test-slave-5rg1 krb5kdc[20730](info): commencing operation
krb5kdc: starting...
W20260928 16:48:57.032392 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:48:57.162925 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35971
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:57.277009 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:57.277391 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:57.277480 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:57.281963 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:57.282075 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:57.282114 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:57.282153 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:57.282207 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:57.288138 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:35971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:57.289472 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:57.290576 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:57.299942 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:57.300036 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:57.299934 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:57.300220 20743 server_base.cc:1034] running on GCE node
I20260928 16:48:57.300969 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:57.302218 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:57.303495 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790614137303476 us; error 39 us; skew 500 ppm
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614137, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:57.306486 20743 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:57.307417 20743 webserver.cc:460] Webserver started at http://127.30.196.62:36587/ using document root <none> and password file <none>
I20260928 16:48:57.308046 20743 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:57.308117 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:57.308291 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:57.310375 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "4670776fa1be4eccb9b9b043ddb26167"
format_stamp: "Formatted at 2026-09-28 16:48:57 on dist-test-slave-5rg1"
server_key: "001d9579096becd40044e40a521c2eb7"
server_key_iv: "2f1d2ec767b35562fb32b9d2ae94ef03"
server_key_version: "encryptionkey@0"
I20260928 16:48:57.310918 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "4670776fa1be4eccb9b9b043ddb26167"
format_stamp: "Formatted at 2026-09-28 16:48:57 on dist-test-slave-5rg1"
server_key: "001d9579096becd40044e40a521c2eb7"
server_key_iv: "2f1d2ec767b35562fb32b9d2ae94ef03"
server_key_version: "encryptionkey@0"
I20260928 16:48:57.314775 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:57.317584 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:57.318907 20743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:48:57.319069 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "4670776fa1be4eccb9b9b043ddb26167"
format_stamp: "Formatted at 2026-09-28 16:48:57 on dist-test-slave-5rg1"
server_key: "001d9579096becd40044e40a521c2eb7"
server_key_iv: "2f1d2ec767b35562fb32b9d2ae94ef03"
server_key_version: "encryptionkey@0"
I20260928 16:48:57.319187 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:57.331586 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:57.335391 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:57.335652 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:57.343853 20743 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:35971
I20260928 16:48:57.343900 20810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:35971 every 8 connection(s)
I20260928 16:48:57.345018 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:57.348187 20811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:57.350174 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20743
I20260928 16:48:57.350324 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:57.350633 31504 external_mini_cluster.cc:1428] Setting key 2a37bf532341c6fe2a6ece207836049d
I20260928 16:48:57.355309 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167: Bootstrap starting.
I20260928 16:48:57.358179 20811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:57.359254 20811 log.cc:824] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:48:57.361560 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167: No bootstrap required, opened a new log
I20260928 16:48:57.365211 20811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } }
I20260928 16:48:57.365540 20811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:57.365633 20811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4670776fa1be4eccb9b9b043ddb26167, State: Initialized, Role: FOLLOWER
I20260928 16:48:57.366088 20811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } }
I20260928 16:48:57.366218 20811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:57.366309 20811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:57.366441 20811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:57.366523 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:57.352146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36335 (local address 127.30.196.62:35971)
0928 16:48:57.352611 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:57.352627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:57.352665 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:48:57.353328 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:57.353346 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:57.353704 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:57.354060 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:57.355026 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:57.356179 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:57.357448 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:57.357980 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:57.358113 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:57.361222 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:57.361227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:57.361242 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:57.361273 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:57.363167 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:57.363772 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:57.363777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:57.363782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:57.363896 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:57.364285 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:57.364288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:57.364289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:57.364698 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:57.364963 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:57.365227 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:48:57.365458 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":155,"threads_started":1}
I20260928 16:48:57.367506 20811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } }
I20260928 16:48:57.367906 20811 leader_election.cc:302] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4670776fa1be4eccb9b9b043ddb26167; no voters:
I20260928 16:48:57.368275 20811 leader_election.cc:288] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:57.368357 20816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:57.368575 20816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [term 1 LEADER]: Becoming Leader. State: Replica: 4670776fa1be4eccb9b9b043ddb26167, State: Running, Role: LEADER
I20260928 16:48:57.368943 20816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } }
I20260928 16:48:57.369706 20811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:57.370165 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:57.370185 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4670776fa1be4eccb9b9b043ddb26167. Latest consensus state: current_term: 1 leader_uuid: "4670776fa1be4eccb9b9b043ddb26167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } } }
I20260928 16:48:57.370299 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:57.370620 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4670776fa1be4eccb9b9b043ddb26167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4670776fa1be4eccb9b9b043ddb26167" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35971 } } }
I20260928 16:48:57.370724 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:57.370738 20820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:57.375129 20820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:57.381044 20820 catalog_manager.cc:1347] Generated new cluster ID: 6f480c08b1554944a820ddbe52209ba8
I20260928 16:48:57.381403 20820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:57.384037 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:57.372614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55419 (local address 127.30.196.62:35971)
0928 16:48:57.372740 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:57.372745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:57.373602 (+ 857us) server_negotiation.cc:407] Connection header received
0928 16:48:57.374027 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:57.374032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:57.374089 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:57.374181 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:57.375054 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:57.375928 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:57.377478 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:57.377868 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:57.377950 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:57.380127 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:57.380131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:57.380133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:57.380162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:57.381954 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:57.382638 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:57.382642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:57.382646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:57.382730 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:57.383089 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:57.383094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:57.383100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:57.383271 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:57.383448 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:57.383773 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 16:48:57.383872 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:48:57.406747 20820 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:57.407850 20820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:57.416042 20820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4670776fa1be4eccb9b9b043ddb26167: Generated new TSK 0
I20260928 16:48:57.416968 20820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:57.430295 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20743
W20260928 16:48:57.436337 20728 connection.cc:535] client connection to 127.30.196.62:35971 recv error: Network error: failed to read from TLS socket (remote: 127.30.196.62:35971): Connection reset by peer (error 104)
2026-09-28T16:48:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20843](info): set up 2 sockets
Sep 28 16:48:57 dist-test-slave-5rg1 krb5kdc[20843](info): commencing operation
krb5kdc: starting...
W20260928 16:48:58.477375 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:58 dist-test-slave-5rg1 krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:48:58.604646 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37573
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:37573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:58.718084 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:58.718418 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:58.718497 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:58.722625 20856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:58.722726 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:58.722767 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:58.722828 20856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:58.722867 20856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:58.728425 20856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:37573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:48:58.729645 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:58.730698 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:58.739477 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:58.739477 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:58.739585 20856 server_base.cc:1034] running on GCE node
W20260928 16:48:58.739477 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:58.740545 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:58.741736 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:58.742937 20856 hybrid_clock.cc:648] HybridClock initialized: now 1790614138742907 us; error 38 us; skew 500 ppm
Sep 28 16:48:58 dist-test-slave-5rg1 krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614138, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:58.745775 20856 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:48:58.746690 20856 webserver.cc:460] Webserver started at http://127.30.196.62:43899/ using document root <none> and password file <none>
I20260928 16:48:58.747260 20856 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:58.747359 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:58.747622 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:58.749760 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "f98a9a49c5214aefa254dc994cff94c6"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-5rg1"
server_key: "37170f1414b5d26615b663e2722a1f05"
server_key_iv: "8b2a029d463f4f91e18bcea64558a0cc"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.750386 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "f98a9a49c5214aefa254dc994cff94c6"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-5rg1"
server_key: "37170f1414b5d26615b663e2722a1f05"
server_key_iv: "8b2a029d463f4f91e18bcea64558a0cc"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.754280 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:48:58.756963 20872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:58.758217 20856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:48:58.758414 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "f98a9a49c5214aefa254dc994cff94c6"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-5rg1"
server_key: "37170f1414b5d26615b663e2722a1f05"
server_key_iv: "8b2a029d463f4f91e18bcea64558a0cc"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.758574 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:58.770344 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:58.773509 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:58.773727 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:58.781023 20856 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:37573
I20260928 16:48:58.781073 20924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:37573 every 8 connection(s)
I20260928 16:48:58.782063 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:58.785085 20925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:58.790812 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6: Bootstrap starting.
I20260928 16:48:58.791509 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20856
I20260928 16:48:58.791651 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:48:58.791970 31504 external_mini_cluster.cc:1428] Setting key 1d3d253e3e9ff84c3f9c49c85800352f
I20260928 16:48:58.793916 20925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:58.794657 20925 log.cc:824] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:58.796974 20925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6: No bootstrap required, opened a new log
I20260928 16:48:58.801342 20925 raft_consensus.cc:348] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } }
I20260928 16:48:58.801532 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:58.793373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42575 (local address 127.30.196.62:37573)
0928 16:48:58.793833 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:58.793847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:58.793882 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:48:58.794603 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:58.794621 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:58.795051 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:58.795438 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:58.795867 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.796840 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.798092 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.798567 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.798692 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:58.799640 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:58.799645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:58.799658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:58.800157 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:58.800215 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:58.800239 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:58.800343 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:58.800555 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 16:48:58.800720 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260928 16:48:58.801573 20925 raft_consensus.cc:374] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:58.801687 20925 raft_consensus.cc:729] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98a9a49c5214aefa254dc994cff94c6, State: Initialized, Role: FOLLOWER
I20260928 16:48:58.802146 20925 consensus_queue.cc:260] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } }
I20260928 16:48:58.802268 20925 raft_consensus.cc:388] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:58.802350 20925 raft_consensus.cc:482] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:58.802508 20925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:58.803807 20925 raft_consensus.cc:504] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } }
I20260928 16:48:58.804144 20925 leader_election.cc:302] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f98a9a49c5214aefa254dc994cff94c6; no voters:
I20260928 16:48:58.804471 20925 leader_election.cc:288] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:58.804577 20930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:58.804816 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:58.804836 20930 raft_consensus.cc:686] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [term 1 LEADER]: Becoming Leader. State: Replica: f98a9a49c5214aefa254dc994cff94c6, State: Running, Role: LEADER
I20260928 16:48:58.805135 20930 consensus_queue.cc:237] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } }
I20260928 16:48:58.805859 20925 sys_catalog.cc:564] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:58.806649 20933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f98a9a49c5214aefa254dc994cff94c6. Latest consensus state: current_term: 1 leader_uuid: "f98a9a49c5214aefa254dc994cff94c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } } }
I20260928 16:48:58.806770 20933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:58.807072 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f98a9a49c5214aefa254dc994cff94c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98a9a49c5214aefa254dc994cff94c6" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37573 } } }
I20260928 16:48:58.807173 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:58.807992 20944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:58.811318 20944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:58.812341 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:58.807233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44315 (local address 127.30.196.62:37573)
0928 16:48:58.807353 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:58.807357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:58.808387 (+ 1030us) server_negotiation.cc:407] Connection header received
0928 16:48:58.808586 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:58.808589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:58.808636 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:58.808720 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:58.809043 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.809703 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.810543 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.810785 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.810889 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:58.811841 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:58.811844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:58.811846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:58.811858 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:58.811883 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:58.811893 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:58.811987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:58.812162 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:48:58.812196 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:58.817233 20944 catalog_manager.cc:1347] Generated new cluster ID: 6446243bc9cb42ac9dbc9c9dc1ae2afe
I20260928 16:48:58.817307 20944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:58.827696 20944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:58.828622 20944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:58.838989 20944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f98a9a49c5214aefa254dc994cff94c6: Generated new TSK 0
I20260928 16:48:58.839936 20944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:58.850687 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20856
W20260928 16:48:58.856288 20841 connection.cc:535] client connection to 127.30.196.62:37573 recv error: Network error: failed to read from TLS socket (remote: 127.30.196.62:37573): Connection reset by peer (error 104)
2026-09-28T16:48:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:58 dist-test-slave-5rg1 krb5kdc[20957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:58 dist-test-slave-5rg1 krb5kdc[20957](info): set up 2 sockets
Sep 28 16:48:58 dist-test-slave-5rg1 krb5kdc[20957](info): commencing operation
krb5kdc: starting...
W20260928 16:48:59.900341 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:48:59 dist-test-slave-5rg1 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:00.030931 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37299
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:37299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:00.144984 20970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:00.145354 20970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:00.145447 20970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:00.149636 20970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:00.149747 20970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:00.149788 20970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:00.149850 20970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:00.149890 20970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:00.155416 20970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:37299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:37299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:00.156592 20970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:00.157655 20970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:49:00.166818 20970 server_base.cc:1034] running on GCE node
W20260928 16:49:00.166770 20978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:00.166739 20975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:00.166751 20976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:00.167634 20970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:00.168859 20970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:00.170068 20970 hybrid_clock.cc:648] HybridClock initialized: now 1790614140170045 us; error 32 us; skew 500 ppm
Sep 28 16:49:00 dist-test-slave-5rg1 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614140, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:00.173030 20970 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:00.173938 20970 webserver.cc:460] Webserver started at http://127.30.196.62:32919/ using document root <none> and password file <none>
I20260928 16:49:00.174556 20970 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:00.174665 20970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:00.174947 20970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:00.177150 20970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "24bbbc7266644155aed6f2c6a0be1d44"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-5rg1"
server_key: "5f135d6c2ccb7e1508a6f269f44aeff2"
server_key_iv: "8f177076e795a0a0ad5e4fc3a480f476"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.177766 20970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "24bbbc7266644155aed6f2c6a0be1d44"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-5rg1"
server_key: "5f135d6c2ccb7e1508a6f269f44aeff2"
server_key_iv: "8f177076e795a0a0ad5e4fc3a480f476"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.181738 20970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:00.184621 20985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:00.186112 20970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:00.186317 20970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "24bbbc7266644155aed6f2c6a0be1d44"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-5rg1"
server_key: "5f135d6c2ccb7e1508a6f269f44aeff2"
server_key_iv: "8f177076e795a0a0ad5e4fc3a480f476"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.186468 20970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:00.191295 20970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:00.194621 20970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:00.194832 20970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:00.202099 20970 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:37299
I20260928 16:49:00.202127 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:37299 every 8 connection(s)
I20260928 16:49:00.203042 20970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:00.205911 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:00.208168 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 20970
I20260928 16:49:00.208309 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:00.208614 31504 external_mini_cluster.cc:1428] Setting key 7539774606e1543f228cd843de60c5d8
I20260928 16:49:00.212119 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44: Bootstrap starting.
I20260928 16:49:00.213822 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:00.210062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57493 (local address 127.30.196.62:37299)
0928 16:49:00.210497 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:00.210513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:00.210546 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:49:00.211231 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:00.211247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:00.211520 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:00.211893 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:00.212671 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:00.212677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:00.212698 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:00.212708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:00.212758 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:00.212778 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:00.212944 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:00.213013 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 16:49:00.213162 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260928 16:49:00.214489 21039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:00.215245 21039 log.cc:824] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:00.216624 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:00.217293 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44: No bootstrap required, opened a new log
I20260928 16:49:00.220459 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:00.218779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58263 (local address 127.30.196.62:37299)
0928 16:49:00.218896 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:00.218900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:00.219055 (+ 155us) server_negotiation.cc:407] Connection header received
0928 16:49:00.219178 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:00.219180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:00.219215 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:00.219281 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:00.220042 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:00.220045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:00.220047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:00.220057 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:00.220082 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:00.220091 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:00.220161 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:00.220273 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 16:49:00.220307 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:49:00.221818 21039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } }
I20260928 16:49:00.222039 21039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:00.222074 21039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24bbbc7266644155aed6f2c6a0be1d44, State: Initialized, Role: FOLLOWER
I20260928 16:49:00.222421 21039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } }
I20260928 16:49:00.222528 21039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:00.222565 21039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:00.222635 21039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:00.223519 21039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } }
I20260928 16:49:00.223891 21039 leader_election.cc:302] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24bbbc7266644155aed6f2c6a0be1d44; no voters:
I20260928 16:49:00.224159 21039 leader_election.cc:288] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:00.224320 21049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:00.224588 21049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [term 1 LEADER]: Becoming Leader. State: Replica: 24bbbc7266644155aed6f2c6a0be1d44, State: Running, Role: LEADER
I20260928 16:49:00.224979 21049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } }
I20260928 16:49:00.225430 21039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:00.226591 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24bbbc7266644155aed6f2c6a0be1d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } } }
I20260928 16:49:00.226742 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:00.227181 21058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:00.227138 21051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bbbc7266644155aed6f2c6a0be1d44. Latest consensus state: current_term: 1 leader_uuid: "24bbbc7266644155aed6f2c6a0be1d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bbbc7266644155aed6f2c6a0be1d44" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 37299 } } }
I20260928 16:49:00.227291 21051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:00.230162 21058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:00.236009 21058 catalog_manager.cc:1347] Generated new cluster ID: f7a306c088584a15a4a548b950911c1a
I20260928 16:49:00.236096 21058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:00.257611 21058 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:00.257964 21058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:00.262468 21058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24bbbc7266644155aed6f2c6a0be1d44: Generated new TSK 0
I20260928 16:49:00.263273 21058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:00.274760 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 20970
2026-09-28T16:49:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:00 dist-test-slave-5rg1 krb5kdc[21070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:00 dist-test-slave-5rg1 krb5kdc[21070](info): set up 2 sockets
Sep 28 16:49:00 dist-test-slave-5rg1 krb5kdc[21070](info): commencing operation
krb5kdc: starting...
W20260928 16:49:01.349248 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:01.479305 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:43797
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:01.591109 21083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:01.591403 21083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:01.591449 21083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:01.595682 21083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:01.595749 21083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:01.595768 21083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:01.595786 21083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:01.595801 21083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:01.601307 21083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:01.602406 21083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:01.603425 21083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:01.612468 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:01.612459 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:01.612561 21083 server_base.cc:1034] running on GCE node
W20260928 16:49:01.612459 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:01.613332 21083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:01.614500 21083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:01.615731 21083 hybrid_clock.cc:648] HybridClock initialized: now 1790614141615704 us; error 40 us; skew 500 ppm
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614141, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:01.628813 21083 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:01.629951 21083 webserver.cc:460] Webserver started at http://127.30.196.62:38173/ using document root <none> and password file <none>
I20260928 16:49:01.630576 21083 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:01.630672 21083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:01.630924 21083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:01.633222 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "0b92b06a95e040cdbcdff7720b245183"
format_stamp: "Formatted at 2026-09-28 16:49:01 on dist-test-slave-5rg1"
server_key: "1b50e3ebd32262e1e1ea9185525ee502"
server_key_iv: "2dbe471b0f8b01cf8616db9b19fb653e"
server_key_version: "encryptionkey@0"
I20260928 16:49:01.633898 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "0b92b06a95e040cdbcdff7720b245183"
format_stamp: "Formatted at 2026-09-28 16:49:01 on dist-test-slave-5rg1"
server_key: "1b50e3ebd32262e1e1ea9185525ee502"
server_key_iv: "2dbe471b0f8b01cf8616db9b19fb653e"
server_key_version: "encryptionkey@0"
I20260928 16:49:01.638175 21083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:01.641093 21098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:01.642652 21083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:49:01.642841 21083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "0b92b06a95e040cdbcdff7720b245183"
format_stamp: "Formatted at 2026-09-28 16:49:01 on dist-test-slave-5rg1"
server_key: "1b50e3ebd32262e1e1ea9185525ee502"
server_key_iv: "2dbe471b0f8b01cf8616db9b19fb653e"
server_key_version: "encryptionkey@0"
I20260928 16:49:01.643038 21083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:01.658356 21083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:01.662137 21083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:01.662364 21083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:01.669982 21083 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:43797
I20260928 16:49:01.670006 21150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:43797 every 8 connection(s)
I20260928 16:49:01.670986 21083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:01.673926 21151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:01.676462 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21083
I20260928 16:49:01.676611 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:01.676916 31504 external_mini_cluster.cc:1428] Setting key 317ac9c1f90848cbcbc0bbaf7874cf28
I20260928 16:49:01.680298 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: Bootstrap starting.
I20260928 16:49:01.682524 21151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:01.683279 21151 log.cc:824] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:01.685338 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: No bootstrap required, opened a new log
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:01.689920 21151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } }
I20260928 16:49:01.690204 21151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:01.690300 21151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b92b06a95e040cdbcdff7720b245183, State: Initialized, Role: FOLLOWER
I20260928 16:49:01.690834 21151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } }
I20260928 16:49:01.691013 21151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:01.691110 21151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:01.691233 21151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:01.691931 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:01.678390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45637 (local address 127.30.196.62:43797)
0928 16:49:01.678855 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:01.678870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:01.678905 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:49:01.679558 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:01.679574 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:01.679996 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:01.680329 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:01.681175 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:01.682163 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:01.683092 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:01.683614 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:01.683865 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:01.686654 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:01.686660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:01.686673 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:01.686701 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:01.688572 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:01.689104 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:01.689109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:01.689114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:01.689230 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:01.689556 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:01.689558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:01.689559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:01.690042 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:01.690238 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:01.690605 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 16:49:01.690936 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":148,"threads_started":1}
I20260928 16:49:01.692375 21151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } }
I20260928 16:49:01.692721 21151 leader_election.cc:302] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b92b06a95e040cdbcdff7720b245183; no voters:
I20260928 16:49:01.693095 21151 leader_election.cc:288] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:01.693230 21157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:01.693470 21157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [term 1 LEADER]: Becoming Leader. State: Replica: 0b92b06a95e040cdbcdff7720b245183, State: Running, Role: LEADER
I20260928 16:49:01.693832 21157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } }
I20260928 16:49:01.694404 21151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:01.695065 21159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b92b06a95e040cdbcdff7720b245183. Latest consensus state: current_term: 1 leader_uuid: "0b92b06a95e040cdbcdff7720b245183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } } }
I20260928 16:49:01.695194 21159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:01.695351 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:01.695537 21158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b92b06a95e040cdbcdff7720b245183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92b06a95e040cdbcdff7720b245183" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 43797 } } }
I20260928 16:49:01.695622 21158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183 [sys.catalog]: This master's current role is: LEADER
W20260928 16:49:01.700026 21172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:01.700107 21172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:01.700284 21164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:01.700994 21164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:01.706892 21164 catalog_manager.cc:1347] Generated new cluster ID: fb3b567592c04b579fea2e73428e029d
I20260928 16:49:01.706965 21164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:01.707939 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:01.697694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35087 (local address 127.30.196.62:43797)
0928 16:49:01.697790 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:01.697795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:01.698178 (+ 383us) server_negotiation.cc:407] Connection header received
0928 16:49:01.698274 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:01.698277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:01.698332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:01.698416 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:01.699399 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:01.700164 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:01.701404 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:01.701729 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:01.701852 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:01.703702 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:01.703707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:01.703710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:01.703741 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:01.706057 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:01.706579 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:01.706582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:01.706584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:01.706689 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:01.707034 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:01.707038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:01.707040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:01.707216 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:01.707293 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:01.707617 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:49:01.707782 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 16:49:01.726584 21164 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:01.728123 21164 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:01.737115 21164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b92b06a95e040cdbcdff7720b245183: Generated new TSK 0
I20260928 16:49:01.737871 21164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:01.752135 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21083
2026-09-28T16:49:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21183](info): set up 2 sockets
Sep 28 16:49:01 dist-test-slave-5rg1 krb5kdc[21183](info): commencing operation
krb5kdc: starting...
W20260928 16:49:02.824198 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:49:02 dist-test-slave-5rg1 krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:02.950392 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36247
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:36247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:03.063455 21196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:03.063818 21196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:03.063910 21196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:03.067965 21196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:03.068063 21196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:03.068105 21196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:03.068162 21196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:03.068197 21196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:03.073510 21196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:36247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:03.074647 21196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:03.075739 21196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:03.084606 21204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:03.084594 21202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:03.084589 21201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:03.085098 21196 server_base.cc:1034] running on GCE node
I20260928 16:49:03.085772 21196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:03.086901 21196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:03.088075 21196 hybrid_clock.cc:648] HybridClock initialized: now 1790614143088039 us; error 43 us; skew 500 ppm
Sep 28 16:49:03 dist-test-slave-5rg1 krb5kdc[21183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614143, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:03.091092 21196 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:03.092109 21196 webserver.cc:460] Webserver started at http://127.30.196.62:39701/ using document root <none> and password file <none>
I20260928 16:49:03.092995 21196 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:03.093110 21196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:03.093359 21196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:03.095453 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "691d079ecbd94f808f9896c15724b258"
format_stamp: "Formatted at 2026-09-28 16:49:03 on dist-test-slave-5rg1"
server_key: "5872e9d62fa27b3dc1ed3a9469db0681"
server_key_iv: "2bfd1f4dcee00f10ec37da4f71ad54a1"
server_key_version: "encryptionkey@0"
I20260928 16:49:03.096123 21196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "691d079ecbd94f808f9896c15724b258"
format_stamp: "Formatted at 2026-09-28 16:49:03 on dist-test-slave-5rg1"
server_key: "5872e9d62fa27b3dc1ed3a9469db0681"
server_key_iv: "2bfd1f4dcee00f10ec37da4f71ad54a1"
server_key_version: "encryptionkey@0"
I20260928 16:49:03.099957 21196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:49:03.102531 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:03.103955 21196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:03.104123 21196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "691d079ecbd94f808f9896c15724b258"
format_stamp: "Formatted at 2026-09-28 16:49:03 on dist-test-slave-5rg1"
server_key: "5872e9d62fa27b3dc1ed3a9469db0681"
server_key_iv: "2bfd1f4dcee00f10ec37da4f71ad54a1"
server_key_version: "encryptionkey@0"
I20260928 16:49:03.104254 21196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:03.122534 21196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:03.125748 21196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:03.125989 21196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:03.133600 21196 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:36247
I20260928 16:49:03.133630 21263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:36247 every 8 connection(s)
I20260928 16:49:03.134648 21196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:03.137066 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21196
I20260928 16:49:03.137223 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:03.137583 31504 external_mini_cluster.cc:1428] Setting key 7258c3fc05885117ebc710be43f12cab
I20260928 16:49:03.137841 21264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:03.144685 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258: Bootstrap starting.
I20260928 16:49:03.147069 21264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:03.147214 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:03.139096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56145 (local address 127.30.196.62:36247)
0928 16:49:03.139576 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:03.139590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:03.139628 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:49:03.140770 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:03.140791 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:03.141177 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:03.141546 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:03.141959 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:03.142962 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:03.143946 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:03.144442 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:03.144637 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:03.145538 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:03.145544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:03.145556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:03.145948 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:03.146004 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:03.146026 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:03.146140 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:03.146255 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 16:49:03.146481 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":164,"threads_started":1}
I20260928 16:49:03.148032 21264 log.cc:824] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:03.150100 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:03.150282 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258: No bootstrap required, opened a new log
I20260928 16:49:03.154699 21264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } }
I20260928 16:49:03.155036 21264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:03.155109 21264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691d079ecbd94f808f9896c15724b258, State: Initialized, Role: FOLLOWER
I20260928 16:49:03.155658 21264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } }
I20260928 16:49:03.155892 21264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:03.155977 21264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:03.156093 21264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:03.156709 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:03.152270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57481 (local address 127.30.196.62:36247)
0928 16:49:03.152410 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:03.152417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:03.152608 (+ 191us) server_negotiation.cc:407] Connection header received
0928 16:49:03.152754 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:03.152756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:03.152797 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:03.152864 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:03.153245 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:03.153784 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:03.154903 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:03.155244 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:03.155341 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:03.156257 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:03.156260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:03.156263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:03.156277 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:03.156308 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:03.156319 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:03.156379 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:03.156512 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:49:03.156562 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:49:03.157536 21264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } }
I20260928 16:49:03.157940 21264 leader_election.cc:302] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691d079ecbd94f808f9896c15724b258; no voters:
I20260928 16:49:03.158283 21264 leader_election.cc:288] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:03.158423 21274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:03.158692 21274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [term 1 LEADER]: Becoming Leader. State: Replica: 691d079ecbd94f808f9896c15724b258, State: Running, Role: LEADER
I20260928 16:49:03.159068 21274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } }
I20260928 16:49:03.159722 21264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:03.160766 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691d079ecbd94f808f9896c15724b258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } } }
I20260928 16:49:03.160905 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:03.161232 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691d079ecbd94f808f9896c15724b258. Latest consensus state: current_term: 1 leader_uuid: "691d079ecbd94f808f9896c15724b258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d079ecbd94f808f9896c15724b258" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 36247 } } }
I20260928 16:49:03.161382 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:03.161438 21283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:03.164495 21283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:03.170470 21283 catalog_manager.cc:1347] Generated new cluster ID: 714fbb3e9e244c82a840d466750b811f
I20260928 16:49:03.170670 21283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:03.179648 21283 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:03.180776 21283 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:03.192528 21283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691d079ecbd94f808f9896c15724b258: Generated new TSK 0
I20260928 16:49:03.193310 21283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:03.205081 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21196
2026-09-28T16:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:03 dist-test-slave-5rg1 krb5kdc[21296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:03 dist-test-slave-5rg1 krb5kdc[21296](info): set up 2 sockets
Sep 28 16:49:03 dist-test-slave-5rg1 krb5kdc[21296](info): commencing operation
krb5kdc: starting...
W20260928 16:49:04.257089 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:49:04 dist-test-slave-5rg1 krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:04.392846 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35163
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:04.505993 21309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:04.506284 21309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:04.506327 21309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:04.510501 21309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:04.510573 21309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:04.510592 21309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:04.510609 21309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:04.510625 21309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:04.518045 21309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:35163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:04.519644 21309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:04.521123 21309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:04.530539 21315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:04.530702 21309 server_base.cc:1034] running on GCE node
W20260928 16:49:04.530539 21314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:04.530560 21317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:04.531450 21309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:04.532675 21309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:04.533890 21309 hybrid_clock.cc:648] HybridClock initialized: now 1790614144533862 us; error 41 us; skew 500 ppm
Sep 28 16:49:04 dist-test-slave-5rg1 krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614144, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:04.536803 21309 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:04.537768 21309 webserver.cc:460] Webserver started at http://127.30.196.62:45511/ using document root <none> and password file <none>
I20260928 16:49:04.538350 21309 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:04.538453 21309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:04.538690 21309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:04.540861 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "48a5d24b16ae48d0a812c398bdb0ec2c"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-5rg1"
server_key: "6dae18f0df565163da5432264cb208e8"
server_key_iv: "fae5ac3f2809ff6cffffeb616785acad"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.541469 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "48a5d24b16ae48d0a812c398bdb0ec2c"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-5rg1"
server_key: "6dae18f0df565163da5432264cb208e8"
server_key_iv: "fae5ac3f2809ff6cffffeb616785acad"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.545305 21309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:49:04.547854 21324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:04.549202 21309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:49:04.549357 21309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "48a5d24b16ae48d0a812c398bdb0ec2c"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-5rg1"
server_key: "6dae18f0df565163da5432264cb208e8"
server_key_iv: "fae5ac3f2809ff6cffffeb616785acad"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.549486 21309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:04.554207 21309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:04.557163 21309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:04.557360 21309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:04.564986 21309 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:35163
I20260928 16:49:04.565002 21376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:35163 every 8 connection(s)
I20260928 16:49:04.566004 21309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:04.568928 21377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:04.569387 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21309
I20260928 16:49:04.569528 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:04.569841 31504 external_mini_cluster.cc:1428] Setting key 478432daf57c7b49f07e180c669822c2
I20260928 16:49:04.575171 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:04.571335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57997 (local address 127.30.196.62:35163)
0928 16:49:04.571835 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:04.571850 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:04.571885 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:49:04.572581 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:04.572598 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:04.572868 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:04.573209 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:04.574037 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:04.574042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:04.574054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:04.574065 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:04.574116 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:04.574135 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:04.574278 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:04.574348 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 16:49:04.574505 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260928 16:49:04.576068 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c: Bootstrap starting.
I20260928 16:49:04.578027 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:04.578550 21377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:04.579326 21377 log.cc:824] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:04.581672 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c: No bootstrap required, opened a new log
I20260928 16:49:04.581869 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:04.580215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50313 (local address 127.30.196.62:35163)
0928 16:49:04.580341 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:04.580346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:04.580539 (+ 193us) server_negotiation.cc:407] Connection header received
0928 16:49:04.580654 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:04.580656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:04.580685 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:04.580748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:04.581453 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:04.581457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:04.581459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:04.581469 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:04.581494 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:04.581505 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:04.581573 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:04.581683 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 16:49:04.581713 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:49:04.585469 21377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } }
I20260928 16:49:04.585729 21377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:04.585790 21377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a5d24b16ae48d0a812c398bdb0ec2c, State: Initialized, Role: FOLLOWER
I20260928 16:49:04.586251 21377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } }
I20260928 16:49:04.586416 21377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:04.586488 21377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:04.586586 21377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:04.587527 21377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } }
I20260928 16:49:04.587896 21377 leader_election.cc:302] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48a5d24b16ae48d0a812c398bdb0ec2c; no voters:
I20260928 16:49:04.588219 21377 leader_election.cc:288] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:04.588353 21387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:04.588586 21387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [term 1 LEADER]: Becoming Leader. State: Replica: 48a5d24b16ae48d0a812c398bdb0ec2c, State: Running, Role: LEADER
I20260928 16:49:04.588907 21387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } }
I20260928 16:49:04.589655 21377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:04.590922 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a5d24b16ae48d0a812c398bdb0ec2c. Latest consensus state: current_term: 1 leader_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } } }
I20260928 16:49:04.590955 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d24b16ae48d0a812c398bdb0ec2c" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 35163 } } }
I20260928 16:49:04.591069 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:04.591068 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:04.591455 21396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:04.594856 21396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:04.600805 21396 catalog_manager.cc:1347] Generated new cluster ID: 418eb5f0c09a4a1fa7615060b3694d2e
I20260928 16:49:04.600934 21396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:04.611346 21396 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:04.611732 21396 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:04.623195 21396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48a5d24b16ae48d0a812c398bdb0ec2c: Generated new TSK 0
I20260928 16:49:04.624197 21396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:04.641538 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21309
2026-09-28T16:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1436 ms)
[----------] 18 tests from AuthTokenIssuingTest (25860 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:04 dist-test-slave-5rg1 krb5kdc[21408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:04 dist-test-slave-5rg1 krb5kdc[21408](info): set up 2 sockets
Sep 28 16:49:04 dist-test-slave-5rg1 krb5kdc[21408](info): commencing operation
krb5kdc: starting...
W20260928 16:49:05.688465 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:49:05 dist-test-slave-5rg1 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:05.819424 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38871
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:05.933799 21421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:05.934120 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:05.934145 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:05.934185 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:05.938324 21421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:05.938400 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:05.938421 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:05.938441 21421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:05.938457 21421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:05.943861 21421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:38871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:05.944908 21421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:05.945887 21421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:05.954820 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:05.954754 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:05.954903 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:05.954998 21421 server_base.cc:1034] running on GCE node
I20260928 16:49:05.955753 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:05.956980 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:05.958179 21421 hybrid_clock.cc:648] HybridClock initialized: now 1790614145958128 us; error 62 us; skew 500 ppm
Sep 28 16:49:05 dist-test-slave-5rg1 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:05.961349 21421 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:05.962280 21421 webserver.cc:460] Webserver started at http://127.30.196.62:45389/ using document root <none> and password file <none>
I20260928 16:49:05.962875 21421 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:05.962971 21421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:05.963213 21421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:05.965380 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "f8fd74daa3b940df97b6e8a675401a9f"
format_stamp: "Formatted at 2026-09-28 16:49:05 on dist-test-slave-5rg1"
server_key: "bad2ed9e6db3529987d11b49e61104d8"
server_key_iv: "32c27c14eac6a94473182380a314d047"
server_key_version: "encryptionkey@0"
I20260928 16:49:05.966007 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "f8fd74daa3b940df97b6e8a675401a9f"
format_stamp: "Formatted at 2026-09-28 16:49:05 on dist-test-slave-5rg1"
server_key: "bad2ed9e6db3529987d11b49e61104d8"
server_key_iv: "32c27c14eac6a94473182380a314d047"
server_key_version: "encryptionkey@0"
I20260928 16:49:05.969947 21421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:05.972625 21437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:05.973973 21421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:05.974149 21421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "f8fd74daa3b940df97b6e8a675401a9f"
format_stamp: "Formatted at 2026-09-28 16:49:05 on dist-test-slave-5rg1"
server_key: "bad2ed9e6db3529987d11b49e61104d8"
server_key_iv: "32c27c14eac6a94473182380a314d047"
server_key_version: "encryptionkey@0"
I20260928 16:49:05.974289 21421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:06.004606 21421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:06.008065 21421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:06.008289 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:06.015512 21421 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:38871
I20260928 16:49:06.015842 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:38871 every 8 connection(s)
I20260928 16:49:06.016664 21421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:06.019526 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:06.023538 21490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.026775 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21421
I20260928 16:49:06.026919 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:06.027209 31504 external_mini_cluster.cc:1428] Setting key 90f8c7b4479978b3adfb3163cc3b2ef2
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.61@KRBTEST.COM" created.
I20260928 16:49:06.040601 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.025842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38871 (local address 127.0.0.1:35096)
0928 16:49:06.026493 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.026547 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.027643 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.028378 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.028386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.028904 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.029255 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.029269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.031035 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.031038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.031937 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.031947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.032993 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.032997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.033097 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.033736 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.033754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.035220 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.037357 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.037366 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.037374 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.037629 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.037924 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.037926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.037927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.038108 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.038705 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.038722 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.038851 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.039094 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.039512 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":118,"spinlock_wait_cycles":21888,"thread_start_us":159,"threads_started":1}
I20260928 16:49:06.040601 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.025842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.30.196.62:38871)
0928 16:49:06.026370 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.026389 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.026680 (+ 291us) server_negotiation.cc:407] Connection header received
0928 16:49:06.027996 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.028000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.028170 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.028313 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.029416 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.030883 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.032063 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.032739 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.032954 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.035381 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.035390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.035405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.035438 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.037248 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.037731 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.037735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.037736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.037842 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.038180 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.038182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.038183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.038577 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.038684 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.039157 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.039430 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":138,"threads_started":1}
I20260928 16:49:06.043066 21490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.043876 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.043575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57736)
0928 16:49:06.043730 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.043810 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:49:06.044081 21439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
I20260928 16:49:06.045006 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.044785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57750)
0928 16:49:06.044895 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.044940 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:49:06.045136 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:37811: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
I20260928 16:49:06.055313 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:37811
--webserver_interface=127.30.196.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:06.094390 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } attempt: 1
I20260928 16:49:06.095279 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.094983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57758)
0928 16:49:06.095134 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.095206 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 16:49:06.096196 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.095955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57762)
0928 16:49:06.096075 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.096135 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:49:06.096411 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:37811: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
I20260928 16:49:06.164588 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } attempt: 2
I20260928 16:49:06.165499 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.165168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57766)
0928 16:49:06.165325 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.165396 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:49:06.166342 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.166116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57776)
0928 16:49:06.166235 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.166283 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:49:06.166548 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:37811: Network error: Client connection negotiation failed: client connection to 127.30.196.61:37811: connect: Connection refused (error 111)
W20260928 16:49:06.169162 21497 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:06.169494 21497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:06.169541 21497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:06.169621 21497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:06.173764 21497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:06.173820 21497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:06.173844 21497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:06.173862 21497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:06.173877 21497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:06.179466 21497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.196.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:06.180687 21497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:06.181728 21497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:06.190686 21503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:06.190733 21502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:06.190709 21505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:06.191097 21497 server_base.cc:1034] running on GCE node
I20260928 16:49:06.191861 21497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:06.193055 21497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:06.194275 21497 hybrid_clock.cc:648] HybridClock initialized: now 1790614146194238 us; error 56 us; skew 500 ppm
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:06.197381 21497 init.cc:377] Logged in from keytab as kudu/127.30.196.61@KRBTEST.COM (short username kudu)
I20260928 16:49:06.198331 21497 webserver.cc:460] Webserver started at http://127.30.196.61:35467/ using document root <none> and password file <none>
I20260928 16:49:06.198928 21497 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:06.199024 21497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:06.199265 21497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:06.201577 21497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data/instance:
uuid: "45a82ea71818466abb3b87370a819646"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "417ce71cff4fae98bc1c8fcd2f8d3233"
server_key_iv: "526d7590b9ab2cb595b61877b1e881fa"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.202262 21497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal/instance:
uuid: "45a82ea71818466abb3b87370a819646"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "417ce71cff4fae98bc1c8fcd2f8d3233"
server_key_iv: "526d7590b9ab2cb595b61877b1e881fa"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.206144 21497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:49:06.208742 21512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:06.210057 21497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:49:06.210211 21497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "45a82ea71818466abb3b87370a819646"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "417ce71cff4fae98bc1c8fcd2f8d3233"
server_key_iv: "526d7590b9ab2cb595b61877b1e881fa"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.210340 21497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:06.224987 21497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:06.227903 21497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:06.228121 21497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:06.235287 21497 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.61:37811
I20260928 16:49:06.235589 21564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.61:37811 every 8 connection(s)
I20260928 16:49:06.236353 21497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
I20260928 16:49:06.239089 21565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:06.242112 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21497
I20260928 16:49:06.242218 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-1/wal/instance
I20260928 16:49:06.242472 31504 external_mini_cluster.cc:1428] Setting key 6b56cd36d56584b29636a5e705a71819
I20260928 16:49:06.242941 21565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.196.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.60@KRBTEST.COM" created.
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:06.257730 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } attempt: 3
I20260928 16:49:06.261394 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.245143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.30.196.62:38871)
0928 16:49:06.245301 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.245306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.246213 (+ 907us) server_negotiation.cc:407] Connection header received
0928 16:49:06.247126 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.247132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.247197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.247298 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.248715 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.249294 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.251106 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.251366 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.251462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.256875 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.256882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.256888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.256955 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.258818 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.259398 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.259405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.259410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.259506 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.259945 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.259949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.259951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.260197 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.260312 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.260977 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.261122 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:49:06.262504 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.245562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38871 (local address 127.0.0.1:35110)
0928 16:49:06.246063 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.246105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.246914 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.247459 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.247466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.247875 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.248520 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.248547 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.249447 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.249453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.250953 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.250967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.251555 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.251560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.251844 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.252616 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.252642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.256643 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.258991 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.258997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.259020 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.259282 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.259612 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.259615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.259617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.259827 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.260312 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.260319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.260661 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.260862 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.261479 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":210,"thread_start_us":118,"threads_started":1}
I20260928 16:49:06.264520 21565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:06.269767 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.258544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57792)
0928 16:49:06.258778 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.258793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.258871 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.259913 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.259918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.260116 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.260271 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.260282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.261681 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.261685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.262488 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.262502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.263330 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.263333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.263443 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.263932 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.263950 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.265574 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.268022 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.268025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.268026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.268256 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.268574 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.268577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.268579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.268678 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.269174 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.269177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.269284 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.269409 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.269557 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 16:49:06.270176 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.258655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.30.196.61:37811)
0928 16:49:06.259426 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.259437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.259455 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:06.259496 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.259500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.259772 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.259919 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.260403 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.261545 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.262692 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.263204 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.263320 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.265695 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.265703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.265712 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.265730 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.267910 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.268375 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.268379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.268381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.268458 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.268774 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.268779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.268781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.269053 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.269199 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.269518 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.269715 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":246,"thread_start_us":155,"threads_started":1}
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:06.272328 21490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.273066 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.272809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57700)
0928 16:49:06.272928 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.272987 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:49:06.273909 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.273623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57716)
0928 16:49:06.273795 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.273844 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 16:49:06.274097 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.274562 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.265089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57794)
0928 16:49:06.265264 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.265277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.265351 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.266026 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.266028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.266204 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.266353 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.266361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.267379 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.267383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.268176 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.268184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.268799 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.268813 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.268956 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.269480 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.269501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.270960 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.272951 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.272956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.272958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.273143 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.273460 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.273463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.273466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.273569 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.274018 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.274022 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.274122 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.274240 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.274391 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":23,"spinlock_wait_cycles":1536}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
I20260928 16:49:06.274690 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.265082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.30.196.61:37811)
0928 16:49:06.265814 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.265821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.265832 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:49:06.265861 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.265864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.265896 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.266006 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.266500 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.267268 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.268289 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.268684 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.268799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.271114 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.271118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.271120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.271149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.272835 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.273263 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.273268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.273269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.273356 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.273661 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.273663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.273664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.273888 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.274007 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.274296 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.274446 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":672,"thread_start_us":110,"threads_started":1}
I20260928 16:49:06.275594 21565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.275831 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:45467
--webserver_interface=127.30.196.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:06.276278 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.276055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57732)
0928 16:49:06.276147 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.276226 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 16:49:06.276520 21516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.277393 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.277116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57748)
0928 16:49:06.277271 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.277320 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 16:49:06.277585 21565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.305310 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 1
I20260928 16:49:06.306319 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.305967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57762)
0928 16:49:06.306154 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.306222 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 16:49:06.307142 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.306911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57764)
0928 16:49:06.307027 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.307082 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:49:06.307276 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.326905 21565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 1
I20260928 16:49:06.327819 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.327480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57774)
0928 16:49:06.327638 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.327738 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:49:06.328581 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.328354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57784)
0928 16:49:06.328470 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.328524 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:49:06.328737 21565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.382470 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 2
I20260928 16:49:06.383335 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.383025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57790)
0928 16:49:06.383187 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.383254 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:49:06.384174 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.383946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57792)
0928 16:49:06.384061 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.384108 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:49:06.384383 21490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
W20260928 16:49:06.393146 21573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:06.393472 21573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:06.393527 21573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:06.393592 21573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 16:49:06.396921 21565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 2
W20260928 16:49:06.397748 21573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:06.397800 21573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260928 16:49:06.397765 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.397473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57798)
0928 16:49:06.397621 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.397690 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 16:49:06.397820 21573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:06.397837 21573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:06.397877 21573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:06.398569 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.398322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57810)
0928 16:49:06.398432 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.398505 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:49:06.398728 21565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45467: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45467: connect: Connection refused (error 111)
I20260928 16:49:06.403860 21573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:38871,127.30.196.61:37811,127.30.196.60:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:45467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.196.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:06.405027 21573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:06.406112 21573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:06.414866 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:06.414870 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:06.415113 21573 server_base.cc:1034] running on GCE node
W20260928 16:49:06.415301 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:06.415972 21573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:06.417176 21573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:06.418341 21573 hybrid_clock.cc:648] HybridClock initialized: now 1790614146418313 us; error 42 us; skew 500 ppm
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:06.421253 21573 init.cc:377] Logged in from keytab as kudu/127.30.196.60@KRBTEST.COM (short username kudu)
I20260928 16:49:06.422330 21573 webserver.cc:460] Webserver started at http://127.30.196.60:35057/ using document root <none> and password file <none>
I20260928 16:49:06.422928 21573 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:06.423028 21573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:06.423273 21573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:06.425420 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data/instance:
uuid: "b6423ed2c71b4c90a988eec92a6d4f7c"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "520cb7ccd4c705fdc6418aac2304059f"
server_key_iv: "4b25d9afe9f57409a8e85e3aea3b11c5"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.426002 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal/instance:
uuid: "b6423ed2c71b4c90a988eec92a6d4f7c"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "520cb7ccd4c705fdc6418aac2304059f"
server_key_iv: "4b25d9afe9f57409a8e85e3aea3b11c5"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.429872 21573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:49:06.432605 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:06.433957 21573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:06.434122 21573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "b6423ed2c71b4c90a988eec92a6d4f7c"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-5rg1"
server_key: "520cb7ccd4c705fdc6418aac2304059f"
server_key_iv: "4b25d9afe9f57409a8e85e3aea3b11c5"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.434253 21573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:06.455595 21573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:06.459144 21573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:06.459365 21573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:06.466686 21573 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.60:45467
I20260928 16:49:06.466989 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.60:45467 every 8 connection(s)
I20260928 16:49:06.467739 21573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
I20260928 16:49:06.470606 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:06.473235 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21573
I20260928 16:49:06.473372 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613859608463-31504-0/minicluster-data/master-2/wal/instance
I20260928 16:49:06.473636 31504 external_mini_cluster.cc:1428] Setting key 78269de6feed2fd7ec6ba086092e2fb5
I20260928 16:49:06.475102 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:49:06.483549 21490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 3
I20260928 16:49:06.488150 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.476934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.30.196.62:38871)
0928 16:49:06.477075 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.477081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.477416 (+ 335us) server_negotiation.cc:407] Connection header received
0928 16:49:06.477577 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.477580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.477625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.477718 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.478638 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.479256 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.480446 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.480673 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.480795 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.483482 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.483485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.483488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.483553 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.485690 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.486236 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.486243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.486247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.486344 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.486764 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.486768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.486770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.487025 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.487141 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.487814 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.487958 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:49:06.489090 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.476830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:38871 (local address 127.0.0.1:35118)
0928 16:49:06.477269 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.477308 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.477473 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.477771 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.477776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.478264 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.478504 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.478514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.479377 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.479381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.480325 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.480336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.480836 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.480840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.480975 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.481668 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.481684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.483286 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.485808 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.485818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.485833 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.486117 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.486458 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.486461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.486463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.486649 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.487139 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.487147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.487466 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.487733 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.488101 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":5,"thread_start_us":132,"threads_started":1}
I20260928 16:49:06.489566 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.474954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.30.196.60:45467)
0928 16:49:06.475383 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.475398 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.475432 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:49:06.476172 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.476190 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.476567 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.476896 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.478160 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.479236 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.480300 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.480676 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.480851 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.483991 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.484030 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.484047 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.484797 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.487050 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.487559 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.487564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.487566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.487649 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.488002 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.488006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.488008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.488299 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.488508 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.489248 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.489389 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:49:06.489884 21565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } attempt: 3
I20260928 16:49:06.490602 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.495630 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.484039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57818)
0928 16:49:06.484246 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.484265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.484346 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.484709 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.484711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.484871 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.485041 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.485050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.486134 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.486139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.486814 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.486831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.487322 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.487326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.487430 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.488346 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.488364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.490172 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.493938 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.493941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.493943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.494209 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.494566 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.494569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.494571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.494664 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.495081 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.495084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.495200 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.495285 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.495460 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:06.497643 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.484084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.30.196.60:45467)
0928 16:49:06.484437 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.484446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.484472 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:49:06.484509 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.484512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.484564 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.484659 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.485201 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.486029 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.486933 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.487203 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.487303 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.490334 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.490337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.490340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.490364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.493757 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.494343 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.494346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.494349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.494459 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.494778 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.494781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.494783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.494974 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.497309 (+ 2335us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.497373 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.497474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:06.501485 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.490347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57826)
0928 16:49:06.490616 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.490630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.490708 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.490918 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.490920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.491100 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.491238 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.491245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.492695 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.492702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.493841 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.493850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.494444 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.494448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.494544 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.495008 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.495022 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.497143 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.499846 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.499849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.499850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.500175 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.500490 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.500495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.500578 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.501015 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.501019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.501108 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.501185 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.501324 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260928 16:49:06.501686 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.491408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.30.196.61:37811)
0928 16:49:06.491519 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.491524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.491636 (+ 112us) server_negotiation.cc:407] Connection header received
0928 16:49:06.492741 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.492748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.492808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.492865 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.493513 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.494293 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.495429 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.495800 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.495894 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.498107 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.498110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.498114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.498138 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.499833 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500351 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.500355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.500356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.500435 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500748 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.500751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.500752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.501003 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.501071 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.501393 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.501540 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:49:06.502092 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.491014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:37811 (local address 127.0.0.1:57808)
0928 16:49:06.491551 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.491568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.491685 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.493030 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.493037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.493260 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.493358 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.493364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.494419 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.494424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.495300 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.495313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.496141 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.496144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.496209 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.496679 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.496690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.497946 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.499948 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.499954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.499955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.500245 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.500561 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.500565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.500647 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.501141 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.501144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.501226 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.501779 (+ 553us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.501915 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453}
I20260928 16:49:06.502475 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.491827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.30.196.62:38871)
0928 16:49:06.491951 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.491965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.491996 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:49:06.492125 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.492127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.492178 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.492248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.493125 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.493851 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.494978 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.495328 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.495415 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.498839 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.498845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.498851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.498883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.500487 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500983 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.500986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.500988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.501073 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.501414 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.501417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.501419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.501622 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.501806 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.502188 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.502302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:49:06.502902 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:06.503090 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.490572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.30.196.60:45467)
0928 16:49:06.490661 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.490665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.490679 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:49:06.490782 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.490784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.490822 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.490901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.491787 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.492561 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.493983 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.494315 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.495587 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.497828 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.497832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.497835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.497857 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.499493 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500295 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.500299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.500302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.500383 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.500686 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.500692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.500694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.500891 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.502270 (+ 1379us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.502855 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.502971 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 16:49:06.503511 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: Bootstrap starting.
I20260928 16:49:06.506805 21565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: Bootstrap starting.
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614146, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
Sep 28 16:49:06 dist-test-slave-5rg1 krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:06.509771 21565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:06.512087 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.503476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45467 (local address 127.0.0.1:57842)
0928 16:49:06.503619 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:06.503633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:06.503728 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:06.504433 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:06.504437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:06.504616 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:06.504701 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.504707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.505419 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.505425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.505943 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:06.505950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.506402 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.506405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:06.506530 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.506997 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:06.507019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:06.508538 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:06.510486 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.510488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.510491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.510763 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.511066 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:06.511071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:06.511074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:06.511160 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:06.511508 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:06.511511 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:06.511590 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 16:49:06.511750 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 16:49:06.511892 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":17}
I20260928 16:49:06.512225 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:06.512087 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.503691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.30.196.60:45467)
0928 16:49:06.503784 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.503788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.503800 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:49:06.503836 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.503838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.503875 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.503954 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.504816 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.505317 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.506055 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.506299 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.506405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.508668 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.508671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.508674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.508695 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.510372 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.510873 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.510876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.510878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.510984 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.511238 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.511243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.511245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.511408 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.511517 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.511795 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.511950 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":38}
I20260928 16:49:06.513211 21490 log.cc:824] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:06.514000 21565 log.cc:824] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:06.514029 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.504346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.30.196.61:37811)
0928 16:49:06.504439 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.504442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.504453 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:49:06.504486 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.504491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.504531 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.504611 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.505340 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.506015 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.507040 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.507358 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.507468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.510221 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.510225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.510227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.510251 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.511938 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.512445 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.512452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.512453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.512549 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.512869 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.512872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.512874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.513105 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.513277 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.513718 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.513828 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 16:49:06.515811 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: Bootstrap starting.
I20260928 16:49:06.516713 21565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: No bootstrap required, opened a new log
I20260928 16:49:06.518658 21643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:06.519448 21643 log.cc:824] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:06.519647 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:06.521237 21464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f8fd74daa3b940df97b6e8a675401a9f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35130
I20260928 16:49:06.523038 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: No bootstrap required, opened a new log
I20260928 16:49:06.527223 21643 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.527520 21643 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6423ed2c71b4c90a988eec92a6d4f7c, State: Initialized, Role: FOLLOWER
I20260928 16:49:06.527931 21643 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.528553 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.528791 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.529414 21643 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:06.530879 21565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.531119 21565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a82ea71818466abb3b87370a819646, State: Initialized, Role: FOLLOWER
I20260928 16:49:06.531479 21565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.532688 21565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:06.537017 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: No bootstrap required, opened a new log
W20260928 16:49:06.534668 21664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:06.539302 21664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:06.541445 21490 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.541706 21490 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fd74daa3b940df97b6e8a675401a9f, State: Initialized, Role: FOLLOWER
I20260928 16:49:06.542058 21490 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.542673 21678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.543015 21678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.543500 21490 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:06.543596 21464 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:49:06.543746 21464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:06.540511 21573 master_runner.cc:186] Error getting master registration for 127.30.196.62:38871: OK, instance_id { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" instance_seqno: 1790614146013469 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:06.544657 21573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:06.545083 21464 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.546065 21464 leader_election.cc:288] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [CANDIDATE]: Term 1 election: Requested vote from peers 45a82ea71818466abb3b87370a819646 (127.30.196.61:37811), b6423ed2c71b4c90a988eec92a6d4f7c (127.30.196.60:45467)
I20260928 16:49:06.547266 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.547417 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.547927 21539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8fd74daa3b940df97b6e8a675401a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45a82ea71818466abb3b87370a819646"
I20260928 16:49:06.548156 21539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:49:06.549160 21689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:06.549237 21689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:06.549697 21539 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fd74daa3b940df97b6e8a675401a9f in term 1.
W20260928 16:49:06.550300 21677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:06.550364 21677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:06.552179 21615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8fd74daa3b940df97b6e8a675401a9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c"
I20260928 16:49:06.552423 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:06.553753 21615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fd74daa3b940df97b6e8a675401a9f in term 1.
I20260928 16:49:06.555310 21441 leader_election.cc:302] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6423ed2c71b4c90a988eec92a6d4f7c, f8fd74daa3b940df97b6e8a675401a9f; no voters:
I20260928 16:49:06.561287 21497 master_runner.cc:263] No leader master found from master 45a82ea71818466abb3b87370a819646
I20260928 16:49:06.561403 21497 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:06.565048 21678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:06.565315 21678 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [term 1 LEADER]: Becoming Leader. State: Replica: f8fd74daa3b940df97b6e8a675401a9f, State: Running, Role: LEADER
I20260928 16:49:06.568081 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.551890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.30.196.60:45467)
0928 16:49:06.554076 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.554081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.557763 (+ 3682us) server_negotiation.cc:407] Connection header received
0928 16:49:06.557802 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.557804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.557848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.557929 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.558859 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.559681 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.561506 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.561758 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.561875 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.563998 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.564001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.564004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.564029 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.566198 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.566669 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.566672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.566674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.566760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.567060 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.567064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.567067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.567276 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.567397 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.567770 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.567890 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2105}
I20260928 16:49:06.574124 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.557217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.30.196.61:37811)
0928 16:49:06.557344 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.557349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.565077 (+ 7728us) server_negotiation.cc:407] Connection header received
0928 16:49:06.565254 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.565258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.565306 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.565356 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.566204 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.566960 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.568172 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.568514 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.568590 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.570864 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.570868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.570871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.570897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.572417 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.572848 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.572851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.572853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.572940 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.573228 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.573232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.573234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.573419 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.573576 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.573803 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.573922 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:49:06.575874 21678 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } }
I20260928 16:49:06.577142 21421 master_runner.cc:263] No leader master found from master f8fd74daa3b940df97b6e8a675401a9f
I20260928 16:49:06.577203 21421 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:06.578150 21678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fd74daa3b940df97b6e8a675401a9f. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.578276 21678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:06.585924 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.575788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.30.196.62:38871)
0928 16:49:06.576047 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.576053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.576068 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:49:06.576107 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.576109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.576155 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.576227 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.578242 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.578948 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.580027 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:06.580362 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:06.580441 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:06.582265 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.582269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.582272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:06.582300 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.584052 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.584554 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.584558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.584560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.584643 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:06.584925 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:06.584928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:06.584930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:06.585152 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.585254 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.585628 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.585749 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260928 16:49:06.587975 21701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:06.589130 21701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:06.598212 21615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [term 1 FOLLOWER]: Refusing update from remote peer f8fd74daa3b940df97b6e8a675401a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:06.599037 21699 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:06.603178 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fd74daa3b940df97b6e8a675401a9f. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.603308 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.607199 21539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [term 1 FOLLOWER]: Refusing update from remote peer f8fd74daa3b940df97b6e8a675401a9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:06.607885 21699 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:06.616029 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.616169 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.618949 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fd74daa3b940df97b6e8a675401a9f. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.619068 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.620247 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.620378 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:06.620618 21702 mvcc.cc:205] Tried to move back new op lower bound from 7334355544458027008 to 7334355544379400192. Current Snapshot: MvccSnapshot[applied={T|T < 7334355544458027008}]
I20260928 16:49:06.622227 21701 catalog_manager.cc:1347] Generated new cluster ID: f1f48cd040a744d3838870e4dbec0b91
I20260928 16:49:06.622288 21701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:06.630209 21678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.630340 21678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:06.632285 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8fd74daa3b940df97b6e8a675401a9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8fd74daa3b940df97b6e8a675401a9f" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 38871 } } peers { permanent_uuid: "45a82ea71818466abb3b87370a819646" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 37811 } } peers { permanent_uuid: "b6423ed2c71b4c90a988eec92a6d4f7c" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45467 } } }
I20260928 16:49:06.632405 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:06.653108 21701 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:06.655292 21701 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:06.677500 21701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8fd74daa3b940df97b6e8a675401a9f: Generated new TSK 0
I20260928 16:49:06.681090 21701 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:49:06.725893 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.722629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.30.196.60:45467)
0928 16:49:06.722767 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.722772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.724763 (+ 1991us) server_negotiation.cc:407] Connection header received
0928 16:49:06.724941 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.724946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.725020 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.725110 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.725744 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57858: BlockingRecv error: recv got EOF from 127.0.0.1:57858 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 16:49:06.727921 21492 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.723402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.30.196.62:38871)
0928 16:49:06.723543 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.723548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.725123 (+ 1575us) server_negotiation.cc:407] Connection header received
0928 16:49:06.726848 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.726852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.726910 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.726971 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.727776 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35152: BlockingRecv error: recv got EOF from 127.0.0.1:35152 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:49:06.729214 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.724187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.30.196.61:37811)
0928 16:49:06.724298 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.724303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.726633 (+ 2330us) server_negotiation.cc:407] Connection header received
0928 16:49:06.728363 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.728367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.728426 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.728505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.729091 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57838: BlockingRecv error: recv got EOF from 127.0.0.1:57838 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 16:49:06.735464 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.733282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.30.196.61:37811)
0928 16:49:06.733402 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.733406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.734371 (+ 965us) server_negotiation.cc:407] Connection header received
0928 16:49:06.734626 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.734629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.734711 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.734770 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.735336 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57842: BlockingRecv error: recv got EOF from 127.0.0.1:57842 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:49:06.741202 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.739488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.30.196.61:37811)
0928 16:49:06.739602 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.739607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.740200 (+ 593us) server_negotiation.cc:407] Connection header received
0928 16:49:06.740413 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.740417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.740486 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.740544 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.741091 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57844: BlockingRecv error: recv got EOF from 127.0.0.1:57844 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:49:06.746932 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.745643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.30.196.61:37811)
0928 16:49:06.745763 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.745767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.746043 (+ 276us) server_negotiation.cc:407] Connection header received
0928 16:49:06.746221 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.746224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.746270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.746318 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.746818 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57848: BlockingRecv error: recv got EOF from 127.0.0.1:57848 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:49:06.755407 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.753672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.30.196.61:37811)
0928 16:49:06.753847 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.753852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.754396 (+ 544us) server_negotiation.cc:407] Connection header received
0928 16:49:06.754616 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.754620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.754685 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.754738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.755290 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57860: BlockingRecv error: recv got EOF from 127.0.0.1:57860 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 16:49:06.767642 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.766236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.30.196.61:37811)
0928 16:49:06.766362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.766366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.766660 (+ 294us) server_negotiation.cc:407] Connection header received
0928 16:49:06.766822 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.766826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.766894 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.766993 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.767514 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57874: BlockingRecv error: recv got EOF from 127.0.0.1:57874 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:49:06.787272 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.785984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.30.196.61:37811)
0928 16:49:06.786113 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.786117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.786254 (+ 137us) server_negotiation.cc:407] Connection header received
0928 16:49:06.786376 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.786379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.786430 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.786515 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.787103 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57880: BlockingRecv error: recv got EOF from 127.0.0.1:57880 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 16:49:06.823081 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.821662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.30.196.61:37811)
0928 16:49:06.821797 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.821808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.822001 (+ 193us) server_negotiation.cc:407] Connection header received
0928 16:49:06.822073 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.822076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.822133 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.822229 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.822832 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57882: BlockingRecv error: recv got EOF from 127.0.0.1:57882 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 16:49:06.890755 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:06.889267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.30.196.61:37811)
0928 16:49:06.889461 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.889466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.889634 (+ 168us) server_negotiation.cc:407] Connection header received
0928 16:49:06.889994 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.889998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.890058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.890150 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.890645 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57894: BlockingRecv error: recv got EOF from 127.0.0.1:57894 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 16:49:07.022397 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:07.021257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.30.196.61:37811)
0928 16:49:07.021410 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.021416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.021457 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:49:07.021623 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.021627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.021688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.021790 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:07.022269 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57910: BlockingRecv error: recv got EOF from 127.0.0.1:57910 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 16:49:07.282574 21571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:07.281260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.30.196.61:37811)
0928 16:49:07.281440 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.281446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.281568 (+ 122us) server_negotiation.cc:407] Connection header received
0928 16:49:07.281769 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.281772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.281822 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.281923 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:07.282427 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57282: BlockingRecv error: recv got EOF from 127.0.0.1:57282 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:49:07.540804 21664 catalog_manager.cc:1259] Loaded cluster ID: f1f48cd040a744d3838870e4dbec0b91
I20260928 16:49:07.540937 21664 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: loading cluster ID for follower catalog manager: success
I20260928 16:49:07.542673 21664 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: acquiring CA information for follower catalog manager: success
I20260928 16:49:07.544019 21664 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6423ed2c71b4c90a988eec92a6d4f7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:07.551592 21677 catalog_manager.cc:1259] Loaded cluster ID: f1f48cd040a744d3838870e4dbec0b91
I20260928 16:49:07.551699 21677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: loading cluster ID for follower catalog manager: success
I20260928 16:49:07.553483 21677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: acquiring CA information for follower catalog manager: success
I20260928 16:49:07.554719 21677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45a82ea71818466abb3b87370a819646: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:07.801836 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.797102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.30.196.61:37811)
0928 16:49:07.797364 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.797368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.797386 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:07.797506 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.797508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.797642 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.797754 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:07.798138 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.798787 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.799969 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.800295 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.800473 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.800899 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:07.801382 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:07.801491 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.801569 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.801625 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260928 16:49:07.809146 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.805268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.30.196.60:45467)
0928 16:49:07.805511 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.805515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.805529 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:49:07.805680 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.805683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.805802 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.805900 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:07.806132 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.806670 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.807715 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.807953 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.808034 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.808312 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:07.808744 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:07.808852 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.808917 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.808976 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260928 16:49:07.817239 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.812392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.30.196.60:45467)
0928 16:49:07.812509 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.812513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.813091 (+ 578us) server_negotiation.cc:407] Connection header received
0928 16:49:07.813267 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.813271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.813323 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.813374 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:07.813914 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.814686 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.815961 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.816249 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.816364 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.816646 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:07.816764 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:07.816838 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.817011 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.817074 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:49:07.817425 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.812275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.30.196.62:38871)
0928 16:49:07.812547 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.812552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.812835 (+ 283us) server_negotiation.cc:407] Connection header received
0928 16:49:07.813089 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.813093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.813272 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.813386 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:07.813941 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.814748 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.815708 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.816004 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.816103 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.816491 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:07.816924 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:07.816993 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.817161 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.817235 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260928 16:49:07.817487 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.812317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.30.196.61:37811)
0928 16:49:07.812620 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.812624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.812960 (+ 336us) server_negotiation.cc:407] Connection header received
0928 16:49:07.813023 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.813027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.813099 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.813167 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:07.813647 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.814558 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.816009 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.816347 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.816455 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.816795 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:07.816882 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:07.816949 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.817296 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.817360 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260928 16:49:07.821336 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21421
I20260928 16:49:07.828210 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21497
I20260928 16:49:07.834707 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21573
2026-09-28T16:49:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3194 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:07 dist-test-slave-5rg1 krb5kdc[21795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:07 dist-test-slave-5rg1 krb5kdc[21795](info): set up 2 sockets
Sep 28 16:49:07 dist-test-slave-5rg1 krb5kdc[21795](info): commencing operation
krb5kdc: starting...
W20260928 16:49:08.892458 31504 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/test-user.keytab.
Sep 28 16:49:08 dist-test-slave-5rg1 krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.196.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
Entry for principal kudu/127.30.196.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab.
I20260928 16:49:09.022646 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46117
--webserver_interface=127.30.196.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:09.136584 21809 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:09.136914 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:09.136945 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:09.136984 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:09.141081 21809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:09.141146 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:09.141166 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:09.141183 21809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:09.141198 21809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:09.146692 21809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.62.keytab
--principal=kudu/127.30.196.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.62:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.196.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:09.147920 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:09.148955 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:09.158015 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.158015 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.158028 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:09.158365 21809 server_base.cc:1034] running on GCE node
I20260928 16:49:09.158977 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:09.160259 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:09.161497 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790614149161473 us; error 45 us; skew 500 ppm
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:09.164750 21809 init.cc:377] Logged in from keytab as kudu/127.30.196.62@KRBTEST.COM (short username kudu)
I20260928 16:49:09.165647 21809 webserver.cc:460] Webserver started at http://127.30.196.62:42875/ using document root <none> and password file <none>
I20260928 16:49:09.166211 21809 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:09.166273 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:09.166460 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:09.169298 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data/instance:
uuid: "01d1363a759d4b189999400b22290bec"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "5533cde599eff6f5e362649474d4a8da"
server_key_iv: "12e1882072867581b3e69d80ecb08e2d"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.169911 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance:
uuid: "01d1363a759d4b189999400b22290bec"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "5533cde599eff6f5e362649474d4a8da"
server_key_iv: "12e1882072867581b3e69d80ecb08e2d"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.174355 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:49:09.177094 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:09.178544 21809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:49:09.178709 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal
uuid: "01d1363a759d4b189999400b22290bec"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "5533cde599eff6f5e362649474d4a8da"
server_key_iv: "12e1882072867581b3e69d80ecb08e2d"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.178855 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:09.197320 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:09.200835 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:09.201009 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:09.208516 21809 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.62:46117
I20260928 16:49:09.208825 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.62:46117 every 8 connection(s)
I20260928 16:49:09.209745 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:09.212680 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:09.216490 21877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:09.219640 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21809
I20260928 16:49:09.219817 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-0/wal/instance
I20260928 16:49:09.220103 31504 external_mini_cluster.cc:1428] Setting key 7f19e7cfb3c5dcdfc9484ebe5efe82f0
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.196.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.61@KRBTEST.COM" created.
I20260928 16:49:09.233239 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.218786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46117 (local address 127.0.0.1:48152)
0928 16:49:09.219342 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.219392 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.220435 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.221245 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.221249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.221597 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.222114 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.222136 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.223354 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.223360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.224672 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.224686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.225370 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.225378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.225538 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.226190 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.226219 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.227934 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.230055 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.230063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.230071 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.230294 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.230727 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.230730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.230731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.230843 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.231726 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.231732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.231922 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.232124 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.232534 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":137,"spinlock_wait_cycles":57088,"thread_start_us":219,"threads_started":1}
I20260928 16:49:09.233247 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.218786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.30.196.62:46117)
0928 16:49:09.219222 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.219237 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.219520 (+ 283us) server_negotiation.cc:407] Connection header received
0928 16:49:09.220760 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.220769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.221030 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.221233 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.222291 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.223216 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.224794 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.225188 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.225305 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.228093 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.228098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.228107 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.228128 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.229930 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.230439 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.230443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.230445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.230621 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.230956 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.230959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.230960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.231552 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.231713 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.232212 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.232423 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":294,"thread_start_us":160,"threads_started":1}
I20260928 16:49:09.235895 21877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:09.236720 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.236442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36026)
0928 16:49:09.236587 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.236659 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 16:49:09.237003 21827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
I20260928 16:49:09.237896 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.237691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36040)
0928 16:49:09.237801 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.237845 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:49:09.238198 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:38013: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
Entry for principal kudu/127.30.196.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab.
I20260928 16:49:09.252342 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:38013
--webserver_interface=127.30.196.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:09.287457 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } attempt: 1
I20260928 16:49:09.288472 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.288136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36046)
0928 16:49:09.288317 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.288389 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:49:09.289271 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.289035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36054)
0928 16:49:09.289150 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.289204 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:49:09.289441 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:38013: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
I20260928 16:49:09.357638 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } attempt: 2
I20260928 16:49:09.358680 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.358325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36068)
0928 16:49:09.358531 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.358598 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 16:49:09.359477 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.359234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36082)
0928 16:49:09.359357 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.359412 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:49:09.359612 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.61:38013: Network error: Client connection negotiation failed: client connection to 127.30.196.61:38013: connect: Connection refused (error 111)
W20260928 16:49:09.370481 21884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:09.370820 21884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:09.370880 21884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:09.370985 21884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:09.375195 21884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:09.375262 21884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:09.375283 21884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:09.375299 21884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:09.375358 21884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:09.381022 21884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.61.keytab
--principal=kudu/127.30.196.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.61:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.196.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:09.382166 21884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:09.383241 21884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:09.392005 21890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.392005 21889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.392109 21892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:09.392364 21884 server_base.cc:1034] running on GCE node
I20260928 16:49:09.392957 21884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:09.394138 21884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:09.395352 21884 hybrid_clock.cc:648] HybridClock initialized: now 1790614149395332 us; error 32 us; skew 500 ppm
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:09.398466 21884 init.cc:377] Logged in from keytab as kudu/127.30.196.61@KRBTEST.COM (short username kudu)
I20260928 16:49:09.399540 21884 webserver.cc:460] Webserver started at http://127.30.196.61:38337/ using document root <none> and password file <none>
I20260928 16:49:09.400198 21884 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:09.400265 21884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:09.400489 21884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:09.402632 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data/instance:
uuid: "d773fa3be1c8434db3ecea72917aeca4"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "45d76ef6168839e9d3cd9028beae2c17"
server_key_iv: "316d6699de050069abad11e1fdbc1533"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.403236 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal/instance:
uuid: "d773fa3be1c8434db3ecea72917aeca4"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "45d76ef6168839e9d3cd9028beae2c17"
server_key_iv: "316d6699de050069abad11e1fdbc1533"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.407167 21884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:49:09.409832 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:09.411214 21884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:49:09.411367 21884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal
uuid: "d773fa3be1c8434db3ecea72917aeca4"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "45d76ef6168839e9d3cd9028beae2c17"
server_key_iv: "316d6699de050069abad11e1fdbc1533"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.411491 21884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:09.438534 21884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:09.441741 21884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:09.441972 21884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:09.449642 21884 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.61:38013
I20260928 16:49:09.450021 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.61:38013 every 8 connection(s)
I20260928 16:49:09.450695 21884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/data/info.pb
I20260928 16:49:09.450810 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } attempt: 3
I20260928 16:49:09.453797 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:09.458348 21953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:09.460027 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21884
I20260928 16:49:09.460171 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-1/wal/instance
I20260928 16:49:09.460561 31504 external_mini_cluster.cc:1428] Setting key 6ffd44dc3ca213c3f9e7ba029484063d
I20260928 16:49:09.464428 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.451416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36092)
0928 16:49:09.451583 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.451602 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.451745 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.454016 (+ 2271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.454020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.454242 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.454389 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.454398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.455738 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.455741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.456591 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.456604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.457391 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.457395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.457501 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.458028 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.458043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.459521 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.462254 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.462257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.462259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.462514 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.462961 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.462967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.462972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.463082 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.463882 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.463886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.464005 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.464115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.464261 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 16:49:09.465545 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.452157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.30.196.61:38013)
0928 16:49:09.452614 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.452629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.452664 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:49:09.453314 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.453330 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.453687 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.454047 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.454540 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.455550 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.456757 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.457248 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.457404 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.459771 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.459778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.459794 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.459830 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.462133 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.462644 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.462650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.462655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.462841 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.463194 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.463197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.463199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.463753 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.463897 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.464269 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.464587 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260928 16:49:09.467535 21877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:09.468443 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.468110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43878)
0928 16:49:09.468295 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.468355 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:09.469626 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.468996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43882)
0928 16:49:09.469449 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.469519 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.30.196.60@KRBTEST.COM; defaulting to no policy
W20260928 16:49:09.470019 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.472718 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.460316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46117 (local address 127.0.0.1:48162)
0928 16:49:09.460783 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.460831 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.460988 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.461270 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.461276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.461627 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.461741 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.461747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.462637 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.462640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.463557 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.463566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.465057 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.465063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.465229 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.467186 (+ 1957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.467206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.469151 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.471147 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.471156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.471162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.471391 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.471658 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.471661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.471707 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.471793 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.472167 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.472172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.472286 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.472407 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.472570 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":159,"threads_started":1}
I20260928 16:49:09.472712 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.460332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.30.196.62:46117)
0928 16:49:09.460498 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.460505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.460904 (+ 399us) server_negotiation.cc:407] Connection header received
0928 16:49:09.461108 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.461114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.461167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.461239 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.461865 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.462514 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.463727 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.464784 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.464903 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.469320 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.469324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.469327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.469352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.470989 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.471496 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.471499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.471500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.471576 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.471874 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.471876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.471877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.472069 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.472184 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.472493 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.472592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.196.60@KRBTEST.COM" created.
I20260928 16:49:09.474423 21953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:09.482712 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.474946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36096)
0928 16:49:09.475066 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.475080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.475159 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.475397 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.475399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.475573 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.475658 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.475727 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.476475 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.476482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.477095 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.477104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.477555 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.477558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.477648 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.478116 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.478136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.479597 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.481298 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.481301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.481302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.481494 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.481766 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.481772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.481773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.481860 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.482201 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.482203 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.482281 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.482361 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.482579 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 16:49:09.482843 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.475016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.30.196.61:38013)
0928 16:49:09.475121 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.475125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.475138 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:49:09.475261 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.475264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.475306 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.475387 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.475832 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.476353 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.477211 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.477443 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.477556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.479783 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.479785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.479787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.479809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.481192 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.481583 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.481585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.481587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.481657 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.481934 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.481953 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.481954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.482122 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.482195 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.482418 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.482664 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":41}
I20260928 16:49:09.483560 21953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:09.484465 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.484202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43884)
0928 16:49:09.484317 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.484412 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:49:09.485095 21903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.485989 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.485773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43894)
0928 16:49:09.485883 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.485934 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:49:09.486153 21953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
Entry for principal kudu/127.30.196.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab.
I20260928 16:49:09.492744 31504 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6WsIs2/build/debug/bin/kudu
/tmp/dist-test-task6WsIs2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:45495
--webserver_interface=127.30.196.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.196.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:09.501187 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 1
I20260928 16:49:09.502195 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.501852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43902)
0928 16:49:09.502051 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.502116 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:49:09.502943 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.502717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43918)
0928 16:49:09.502830 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.502886 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:49:09.503079 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.535408 21953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 1
I20260928 16:49:09.536453 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.536015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43930)
0928 16:49:09.536235 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.536331 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:49:09.537323 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.537073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43944)
0928 16:49:09.537190 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.537245 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:49:09.537451 21953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.578250 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 2
I20260928 16:49:09.579113 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.578782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43960)
0928 16:49:09.578937 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.579035 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:49:09.579938 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.579738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43962)
0928 16:49:09.579841 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.579886 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:49:09.580190 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.605679 21953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 2
I20260928 16:49:09.606673 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.606350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43974)
0928 16:49:09.606513 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.606581 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:49:09.607478 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.607245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43976)
0928 16:49:09.607370 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.607425 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:49:09.607618 21953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
W20260928 16:49:09.608457 21961 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:09.608785 21961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:09.608861 21961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:09.608934 21961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:09.613301 21961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:09.613389 21961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:09.613425 21961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:09.613484 21961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:09.613523 21961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:09.619104 21961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.196.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.196.62:46117,127.30.196.61:38013,127.30.196.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/krb5kdc/kudu_127.30.196.60.keytab
--principal=kudu/127.30.196.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.196.60:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.196.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:49:09.620296 21961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:09.621336 21961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:09.630163 21967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.630179 21966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.630179 21969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:09.630834 21961 server_base.cc:1034] running on GCE node
I20260928 16:49:09.631402 21961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:09.633016 21961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:09.634220 21961 hybrid_clock.cc:648] HybridClock initialized: now 1790614149634181 us; error 51 us; skew 500 ppm
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:09.638207 21961 init.cc:377] Logged in from keytab as kudu/127.30.196.60@KRBTEST.COM (short username kudu)
I20260928 16:49:09.639398 21961 webserver.cc:460] Webserver started at http://127.30.196.60:41761/ using document root <none> and password file <none>
I20260928 16:49:09.640218 21961 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:09.640287 21961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:09.640525 21961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:09.642652 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data/instance:
uuid: "4bacc1ba686a470c9c136877dcc64b6d"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "a3c14d6da5debafb4386fe78475b5411"
server_key_iv: "8f3cf5e4b8d62ea0b9e28bcd3593e5de"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.643240 21961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal/instance:
uuid: "4bacc1ba686a470c9c136877dcc64b6d"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "a3c14d6da5debafb4386fe78475b5411"
server_key_iv: "8f3cf5e4b8d62ea0b9e28bcd3593e5de"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.647190 21961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:49:09.649940 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:09.651540 21961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:49:09.651764 21961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data,/tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal
uuid: "4bacc1ba686a470c9c136877dcc64b6d"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-5rg1"
server_key: "a3c14d6da5debafb4386fe78475b5411"
server_key_iv: "8f3cf5e4b8d62ea0b9e28bcd3593e5de"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.651906 21961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:09.679382 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 3
I20260928 16:49:09.680297 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.679998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43980)
0928 16:49:09.680143 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.680204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:49:09.681182 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.680872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43984)
0928 16:49:09.681054 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.681111 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 16:49:09.681365 21877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.196.60:45495: Network error: Client connection negotiation failed: client connection to 127.30.196.60:45495: connect: Connection refused (error 111)
I20260928 16:49:09.682670 21961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:09.685779 21961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:09.685942 21961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:09.693202 21961 rpc_server.cc:307] RPC server started. Bound to: 127.30.196.60:45495
I20260928 16:49:09.693532 22028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.196.60:45495 every 8 connection(s)
I20260928 16:49:09.694240 21961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/data/info.pb
I20260928 16:49:09.697144 22029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:09.698818 21953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 3
I20260928 16:49:09.700309 31504 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6WsIs2/build/debug/bin/kudu as pid 21961
I20260928 16:49:09.700450 31504 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6WsIs2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613859608463-31504-0/minicluster-data/master-2/wal/instance
I20260928 16:49:09.700786 31504 external_mini_cluster.cc:1428] Setting key 89eb67478ff490d169acd4526d717e3b
I20260928 16:49:09.701745 22029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.61@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:09.714066 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.699386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43986)
0928 16:49:09.699554 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.699572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.699723 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.702623 (+ 2900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.702626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.702840 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.702960 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.702968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.704812 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.704816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.705365 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.705376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.706195 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.706199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.706295 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.706899 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.706921 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.708372 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.711577 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.711581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.711583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.711890 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.712269 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.712273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.712275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.712365 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.713465 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.713468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.713585 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.713703 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.713858 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 16:49:09.715304 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.700177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.30.196.60:45495)
0928 16:49:09.700789 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.700811 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.700858 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:49:09.701753 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.701781 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.702274 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.703081 (+ 807us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.703181 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.704661 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.705502 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.706056 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.706290 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.709077 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.709084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.709096 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.709125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.711448 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.712037 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.712043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.712050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.712168 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.712479 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.712482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.712485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.713349 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.713480 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.713844 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.714185 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":451,"thread_start_us":140,"threads_started":1}
I20260928 16:49:09.715308 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.701500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.30.196.60:45495)
0928 16:49:09.701849 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.701857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.702043 (+ 186us) server_negotiation.cc:407] Connection header received
0928 16:49:09.702186 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.702189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.702274 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.702642 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.703522 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.704659 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.705518 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.706067 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.706628 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.709256 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.709259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.709262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.709296 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.711496 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.712076 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.712079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.712081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.712177 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.712582 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.712585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.712586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.713338 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.713524 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.714663 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.714777 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":515968,"thread_start_us":138,"threads_started":1}
I20260928 16:49:09.715947 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.703889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.30.196.62:46117)
0928 16:49:09.704018 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.704022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.704384 (+ 362us) server_negotiation.cc:407] Connection header received
0928 16:49:09.704598 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.704602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.704656 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.704771 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.706455 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.707167 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.708524 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.708817 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.708939 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.712392 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.712398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.712405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.712441 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.713842 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.714364 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.714368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.714369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.714498 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.714829 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.714832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.714834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.715043 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.715116 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.715644 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.715802 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:49:09.716539 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.703820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.62:46117 (local address 127.0.0.1:48168)
0928 16:49:09.704249 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.704286 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.704470 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.705594 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.705599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.706072 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.706304 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.706315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.707324 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.707330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.708377 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.708387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.708989 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.708994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.709097 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.709640 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.709682 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.712212 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.713946 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.713952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.713958 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.714246 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.714620 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.714623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.714625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.714710 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.715230 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.715236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.715356 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.716143 (+ 787us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.716345 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":28,"thread_start_us":109,"threads_started":1}
I20260928 16:49:09.717530 22029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:09.720237 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: Bootstrap starting.
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:09.722605 21953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:09.723497 21953 log.cc:824] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:09.725847 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: No bootstrap required, opened a new log
I20260928 16:49:09.726253 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.718192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.30.196.61:38013)
0928 16:49:09.718318 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.718322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.718362 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:49:09.718395 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.718397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.718438 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.718520 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.718857 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.719603 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.720425 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.720666 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.720780 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.722749 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.722753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.722755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.722779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.724472 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.724954 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.724961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.724965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.725055 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.725355 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.725358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.725360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.725577 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.725690 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.725981 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.726098 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:49:09.726253 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.717994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.61:38013 (local address 127.0.0.1:36102)
0928 16:49:09.718214 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.718230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.718300 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.718533 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.718535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.718662 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.718751 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.718756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.719756 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.719760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.720293 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.720326 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.720780 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.720783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.720879 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.721389 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.721403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.722612 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.724590 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.724594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.724596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.724836 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.725159 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.725162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.725164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.725254 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.725681 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.725684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.725784 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.725890 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.726107 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 16:49:09.727727 22029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.62@KRBTEST.COM
I20260928 16:49:09.729866 21953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.730085 21953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d773fa3be1c8434db3ecea72917aeca4, State: Initialized, Role: FOLLOWER
I20260928 16:49:09.730584 21953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.731570 22036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.731786 22036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.732162 21953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:09.734150 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.718006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.30.196.62:46117)
0928 16:49:09.718129 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.718135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.723957 (+ 5822us) server_negotiation.cc:407] Connection header received
0928 16:49:09.724071 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.724074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.724125 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.724216 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.725051 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.725770 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.726849 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.727144 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.727229 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.730402 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.730406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.730408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.730440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.732217 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.732722 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.732727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.732729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.732853 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.733134 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.733140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.733143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.733389 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.733633 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.733837 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.733958 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.60@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
W20260928 16:49:09.738588 22047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:09.738736 22047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:09.739024 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.728246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:43990)
0928 16:49:09.728456 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.728472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.728556 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.728917 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.728920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.729091 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.729189 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.729196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.730237 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.730240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.730981 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.730993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.732380 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.732385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.732485 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.733188 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.733205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.735186 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.737495 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.737498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.737501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.737735 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.737983 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.737985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.737987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.738060 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.738543 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.738547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.738622 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.738711 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.738846 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260928 16:49:09.739548 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.728311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.30.196.60:45495)
0928 16:49:09.728701 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.728707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.728720 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:49:09.728749 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.728752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.728786 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.728872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.729322 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.730115 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.731106 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.731410 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.731521 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.735321 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.735325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.735328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.735350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.737368 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.737831 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.737834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.737836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.737903 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.738145 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.738151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.738154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.738404 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.738510 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.739272 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.739394 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260928 16:49:09.736670 21884 master_runner.cc:186] Error getting master registration for 127.30.196.62:46117: OK, instance_id { permanent_uuid: "01d1363a759d4b189999400b22290bec" instance_seqno: 1790614149206419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:09.740509 21884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:09.743291 22029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: Bootstrap starting.
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.196.61@KRBTEST.COM
I20260928 16:49:09.746762 22029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:09.747718 22029 log.cc:824] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:09.748188 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.736210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.30.196.61:38013)
0928 16:49:09.736336 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.736340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.736373 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:49:09.736417 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.736422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.736481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.736552 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.740651 (+ 4099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.741164 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.741968 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.742195 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.742305 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.744866 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.744870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.744873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.744905 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.746255 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.746698 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.746702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.746704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.746786 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.747102 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.747106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.747107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.747295 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.747498 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.747734 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.748053 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:49:09.750433 22029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: No bootstrap required, opened a new log
I20260928 16:49:09.751891 31504 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:09.753067 21851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "01d1363a759d4b189999400b22290bec"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48182
I20260928 16:49:09.755249 22029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.755537 22029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bacc1ba686a470c9c136877dcc64b6d, State: Initialized, Role: FOLLOWER
I20260928 16:49:09.755909 22029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.756428 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.756584 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.757372 22029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:49:09.762802 22061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:09.762892 22061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:09.761227 21961 master_runner.cc:186] Error getting master registration for 127.30.196.62:46117: OK, instance_id { permanent_uuid: "01d1363a759d4b189999400b22290bec" instance_seqno: 1790614149206419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:09.764452 21961 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:09.845580 21877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } attempt: 4
Sep 28 16:49:09 dist-test-slave-5rg1 krb5kdc[21795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.196.62@KRBTEST.COM for kudu/127.30.196.60@KRBTEST.COM
I20260928 16:49:09.855695 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.846342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.30.196.60:45495)
0928 16:49:09.846487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.846493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.846567 (+ 74us) server_negotiation.cc:407] Connection header received
0928 16:49:09.846755 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.846758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.846810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.846887 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.847414 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.848254 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.849100 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.849454 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.849550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.851968 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.851971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.851973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.851997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.853601 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.854124 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.854127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.854129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.854219 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.854633 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.854639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.854641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.854902 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.855000 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.855340 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.855477 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:49:09.855700 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.846259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.196.60:45495 (local address 127.0.0.1:44004)
0928 16:49:09.846486 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:09.846502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:09.846625 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:09.846912 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:09.846915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:09.847146 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:09.847272 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.847278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.848371 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.848374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.848967 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:09.848977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.849557 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.849561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:09.849658 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.850281 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:09.850294 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:09.851816 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:09.853728 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.853733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.853736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.854002 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.854330 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:09.854335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:09.854338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:09.854515 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:09.855013 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:09.855019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:09.855125 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 16:49:09.855243 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:49:09.855485 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260928 16:49:09.859887 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: Bootstrap starting.
I20260928 16:49:09.862857 21877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:09.863860 21877 log.cc:824] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:09.866158 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: No bootstrap required, opened a new log
I20260928 16:49:09.869380 21877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.869633 21877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d1363a759d4b189999400b22290bec, State: Initialized, Role: FOLLOWER
I20260928 16:49:09.869987 21877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.870572 22064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.870858 22064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.871452 21877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:09.871546 21851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:49:09.871637 21851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:09.873152 21851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.875413 21926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d1363a759d4b189999400b22290bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d773fa3be1c8434db3ecea72917aeca4"
I20260928 16:49:09.875923 22003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01d1363a759d4b189999400b22290bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4bacc1ba686a470c9c136877dcc64b6d"
I20260928 16:49:09.876235 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:49:09.877522 22075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:09.877593 22075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:09.877570 22003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d1363a759d4b189999400b22290bec in term 1.
I20260928 16:49:09.878736 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:09.878791 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d1363a759d4b189999400b22290bec, 4bacc1ba686a470c9c136877dcc64b6d; no voters:
I20260928 16:49:09.880152 21926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d1363a759d4b189999400b22290bec in term 1.
I20260928 16:49:09.874220 21851 leader_election.cc:288] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [CANDIDATE]: Term 1 election: Requested vote from peers d773fa3be1c8434db3ecea72917aeca4 (127.30.196.61:38013), 4bacc1ba686a470c9c136877dcc64b6d (127.30.196.60:45495)
I20260928 16:49:09.886550 21809 master_runner.cc:263] No leader master found from master 01d1363a759d4b189999400b22290bec
I20260928 16:49:09.886626 21809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:09.887480 22064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:09.887814 22064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [term 1 LEADER]: Becoming Leader. State: Replica: 01d1363a759d4b189999400b22290bec, State: Running, Role: LEADER
I20260928 16:49:09.888254 22064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } }
I20260928 16:49:09.890447 22064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d1363a759d4b189999400b22290bec. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.890730 22064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:09.893689 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.883021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.30.196.62:46117)
0928 16:49:09.883138 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.883142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.884276 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 16:49:09.884480 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.884484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.884542 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.884594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.885653 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.886394 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.887550 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.887911 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.888003 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.889947 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.889951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.889954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.889981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.891746 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.892270 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.892277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.892280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.892368 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.892684 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.892687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.892689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.892920 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.893042 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.893415 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.893535 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:49:09.906178 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.887012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.30.196.61:38013)
0928 16:49:09.887120 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.887124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.894255 (+ 7131us) server_negotiation.cc:407] Connection header received
0928 16:49:09.894453 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.894458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.894519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.894576 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.895528 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.896344 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.897526 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.897880 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.897979 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.900265 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.900270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.900273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.900310 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.904114 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.904653 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.904657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.904659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.904772 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.905142 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.905145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.905147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.905375 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.905492 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.905875 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.906011 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:49:09.906229 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.883760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.30.196.60:45495)
0928 16:49:09.884040 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.884045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.891952 (+ 7907us) server_negotiation.cc:407] Connection header received
0928 16:49:09.892157 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.892160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.892211 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.892263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.893123 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.893852 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.899814 (+ 5962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.900180 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.900273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.901991 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.901998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.902001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:09.902034 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.903842 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.904391 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.904397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.904400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.904492 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:09.904807 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:09.904811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:09.904812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:09.905074 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.905211 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.905908 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.906063 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260928 16:49:09.906804 22086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:09.908000 22086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:09.914179 22003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [term 1 FOLLOWER]: Refusing update from remote peer 01d1363a759d4b189999400b22290bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:09.914222 21926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [term 1 FOLLOWER]: Refusing update from remote peer 01d1363a759d4b189999400b22290bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:09.915174 22084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:09.915447 22064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:09.919329 22036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d1363a759d4b189999400b22290bec. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.919484 22036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.919641 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d1363a759d4b189999400b22290bec. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.919801 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.926385 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.926558 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:09.926690 22087 mvcc.cc:205] Tried to move back new op lower bound from 7334355558040186880 to 7334355557946949632. Current Snapshot: MvccSnapshot[applied={T|T < 7334355558040186880}]
I20260928 16:49:09.926709 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.926827 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.928393 22089 mvcc.cc:205] Tried to move back new op lower bound from 7334355558040186880 to 7334355557946949632. Current Snapshot: MvccSnapshot[applied={T|T < 7334355558040186880}]
I20260928 16:49:09.935081 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.935223 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:09.936542 22036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d1363a759d4b189999400b22290bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d1363a759d4b189999400b22290bec" member_type: VOTER last_known_addr { host: "127.30.196.62" port: 46117 } } peers { permanent_uuid: "d773fa3be1c8434db3ecea72917aeca4" member_type: VOTER last_known_addr { host: "127.30.196.61" port: 38013 } } peers { permanent_uuid: "4bacc1ba686a470c9c136877dcc64b6d" member_type: VOTER last_known_addr { host: "127.30.196.60" port: 45495 } } }
I20260928 16:49:09.936672 22036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:09.937345 22088 mvcc.cc:205] Tried to move back new op lower bound from 7334355558040186880 to 7334355557946949632. Current Snapshot: MvccSnapshot[applied={T|T < 7334355558040186880}]
I20260928 16:49:09.940145 22086 catalog_manager.cc:1347] Generated new cluster ID: de0f44df58cb446b90407ddf55a1553a
I20260928 16:49:09.940225 22086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:09.982616 22086 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:09.983908 22086 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:09.992084 22086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d1363a759d4b189999400b22290bec: Generated new TSK 0
I20260928 16:49:09.993680 22086 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:49:10.021503 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.020155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.30.196.61:38013)
0928 16:49:10.020283 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.020287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.020507 (+ 220us) server_negotiation.cc:407] Connection header received
0928 16:49:10.020676 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.020679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.020730 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.020792 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.021362 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36136: BlockingRecv error: recv got EOF from 127.0.0.1:36136 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:49:10.021548 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.019769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.30.196.62:46117)
0928 16:49:10.019913 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.019918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.020419 (+ 501us) server_negotiation.cc:407] Connection header received
0928 16:49:10.020486 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.020489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.020552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.020646 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.021406 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48202: BlockingRecv error: recv got EOF from 127.0.0.1:48202 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 16:49:10.021744 22031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.020215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.30.196.60:45495)
0928 16:49:10.020376 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.020381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.020735 (+ 354us) server_negotiation.cc:407] Connection header received
0928 16:49:10.020934 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.020939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.020993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.021051 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.021614 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44022: BlockingRecv error: recv got EOF from 127.0.0.1:44022 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 16:49:10.025199 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.024154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.30.196.61:38013)
0928 16:49:10.024264 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.024270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.024397 (+ 127us) server_negotiation.cc:407] Connection header received
0928 16:49:10.024514 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.024517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.024602 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.024695 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.025092 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36150: BlockingRecv error: recv got EOF from 127.0.0.1:36150 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:49:10.029382 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.028554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.30.196.61:38013)
0928 16:49:10.028686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.028690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.028703 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:49:10.028747 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.028750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.028795 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.028893 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.029280 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36158: BlockingRecv error: recv got EOF from 127.0.0.1:36158 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:49:10.034282 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.033174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.30.196.61:38013)
0928 16:49:10.033308 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.033313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.033553 (+ 240us) server_negotiation.cc:407] Connection header received
0928 16:49:10.033641 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.033644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.033700 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.033790 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.034176 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36174: BlockingRecv error: recv got EOF from 127.0.0.1:36174 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:49:10.041178 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.040040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.30.196.61:38013)
0928 16:49:10.040261 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.040268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.040289 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:49:10.040405 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.040410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.040479 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.040576 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.041061 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36176: BlockingRecv error: recv got EOF from 127.0.0.1:36176 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260928 16:49:10.052356 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.051228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.30.196.61:38013)
0928 16:49:10.051375 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.051380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.051488 (+ 108us) server_negotiation.cc:407] Connection header received
0928 16:49:10.051625 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.051629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.051720 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.051823 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.052232 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36184: BlockingRecv error: recv got EOF from 127.0.0.1:36184 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:49:10.071969 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.070575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.30.196.61:38013)
0928 16:49:10.070787 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.070792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.070968 (+ 176us) server_negotiation.cc:407] Connection header received
0928 16:49:10.071097 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.071099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.071153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.071228 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.071835 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36200: BlockingRecv error: recv got EOF from 127.0.0.1:36200 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 16:49:10.107623 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.106246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.30.196.61:38013)
0928 16:49:10.106411 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.106416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.106607 (+ 191us) server_negotiation.cc:407] Connection header received
0928 16:49:10.106725 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.106728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.106786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.106867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.107485 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36204: BlockingRecv error: recv got EOF from 127.0.0.1:36204 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 16:49:10.175453 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.174232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.30.196.61:38013)
0928 16:49:10.174383 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.174389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.174568 (+ 179us) server_negotiation.cc:407] Connection header received
0928 16:49:10.174693 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.174696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.174746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.174820 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.175281 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36206: BlockingRecv error: recv got EOF from 127.0.0.1:36206 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 16:49:10.307114 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.305766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.30.196.61:38013)
0928 16:49:10.305954 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.305960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.306134 (+ 174us) server_negotiation.cc:407] Connection header received
0928 16:49:10.306313 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.306316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.306374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.306478 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.306970 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36220: BlockingRecv error: recv got EOF from 127.0.0.1:36220 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 16:49:10.566627 21955 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:10.565580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.30.196.61:38013)
0928 16:49:10.565731 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:10.565735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:10.565807 (+ 72us) server_negotiation.cc:407] Connection header received
0928 16:49:10.565936 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:10.565939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:10.565989 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:10.566074 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:10.566481 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36236: BlockingRecv error: recv got EOF from 127.0.0.1:36236 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:49:10.740315 22047 catalog_manager.cc:1259] Loaded cluster ID: de0f44df58cb446b90407ddf55a1553a
I20260928 16:49:10.740453 22047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: loading cluster ID for follower catalog manager: success
I20260928 16:49:10.742120 22047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: acquiring CA information for follower catalog manager: success
I20260928 16:49:10.743361 22047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d773fa3be1c8434db3ecea72917aeca4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:10.764328 22061 catalog_manager.cc:1259] Loaded cluster ID: de0f44df58cb446b90407ddf55a1553a
I20260928 16:49:10.764420 22061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: loading cluster ID for follower catalog manager: success
I20260928 16:49:10.766091 22061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: acquiring CA information for follower catalog manager: success
I20260928 16:49:10.767222 22061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bacc1ba686a470c9c136877dcc64b6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:11.085282 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.081011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.30.196.61:38013)
0928 16:49:11.081284 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.081291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.081310 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:49:11.081522 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.081525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.081653 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.081775 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:11.082108 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.082738 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.083638 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.083952 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.084071 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:11.084410 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:11.084866 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:11.085014 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.085055 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.085114 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":66,"threads_started":1}
I20260928 16:49:11.092312 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.088453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.30.196.60:45495)
0928 16:49:11.088701 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.088705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.088718 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:49:11.088815 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.088817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.088937 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.089072 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:11.089284 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.089795 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.090626 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.090848 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.090926 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:11.091251 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:11.091754 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:11.091853 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.092039 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.092120 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260928 16:49:11.099900 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.095773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.30.196.61:38013)
0928 16:49:11.095911 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.095916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.096088 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:49:11.096207 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.096212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.096274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.096348 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:11.096728 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.097565 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.098582 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.098884 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.098977 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:11.099345 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:11.099438 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:11.099488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.099697 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.099759 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:49:11.100225 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.096296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.30.196.60:45495)
0928 16:49:11.096418 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.096421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.096650 (+ 229us) server_negotiation.cc:407] Connection header received
0928 16:49:11.096819 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.096823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.096872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.096947 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:11.097199 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.097903 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.098968 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.099236 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.099335 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:11.099650 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:11.099750 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:11.099807 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.100026 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.100104 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:49:11.100750 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.095719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.30.196.62:46117)
0928 16:49:11.096197 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.096202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.096214 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:49:11.096308 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.096314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.096482 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.096604 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:11.096866 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.097528 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.098784 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:11.099116 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:11.099216 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:11.099622 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:11.100011 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:11.100497 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.100534 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.100594 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":73,"threads_started":1}
I20260928 16:49:11.103969 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21809
I20260928 16:49:11.111491 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21884
I20260928 16:49:11.117993 31504 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6WsIs2/build/debug/bin/kudu with pid 21961
2026-09-28T16:49:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3283 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6478 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291501 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST